Ask Claude Code, or any coding agent, to "check this module for vulnerabilities" and you know how it goes. It skims a couple of controllers, flags string concatenation in a test file, writes a confident three-screen report, and an hour later you can't tell which requirements it actually checked and which it quietly skipped. Ask again tomorrow and you get a different report.
I wanted something else: an audit where the agent sees everything a human AppSec engineer would look at (the entry point map, Semgrep and Trivy output, the company's own security standard, the dependency report, past decisions) and where the result lands in a database, tied to a commit, and goes stale when the code changes.
I didn't find a tool that did that, so I built aiegis. It's an Electron and TypeScript desktop app, plus a headless service for Linux, that drives your local coding agent and turns its output into a managed process. MIT licensed: github.com/seelentov/aiegis.
What was wrong with the existing options
SAST tools like Semgrep and CodeQL are deterministic, which is their strength. They also don't know that an endpoint sits behind an auth middleware two layers up, and they can't check code against an internal rule like "every admin action writes an audit log entry". You get hundreds of signals and someone has to triage them by hand.
Agents reason well but live inside one session. The agent has no memory that the billing module was checked last week against the same forty requirements, no list of suppressed false positives, no idea which verdicts went stale after yesterday's merge. Every run starts from zero and you paste context into the prompt by hand.
What I was missing was the layer in between: data in one place, the agent reading it through tools on demand, and writing its results back there.
How it works
aiegis has no chat and no model of its own. It runs claude -p (Claude Code) or pi in the background, inside a separate git worktree on a pinned commit, so your working tree is never touched. The agent talks to the app through a built-in MCP server. Each run gets its own token, and the server knows the run's role and scope and rejects writes outside it: an agent checking the auth module can't write a verdict for payments.
The core object is a matrix of requirements × modules. Requirements come from documents (Markdown, PDF, DOCX); the AI extracts individual items with quotes and a human reviews them. Load OWASP ASVS, your own standard, or let an agent draft a document from the code. Modules are sets of globs, defined by hand or with AI help.
Then there are two passes. A cheap model (Haiku by default) triages cells that don't apply: password storage rules for a module that renders PDFs get n/a with a reason. Sonnet checks the rest and sets pass, finding or needs_human with file references. The app always shows a token estimate before a run, because a single button can cost more than you'd guess.
Some rules live in the MCP server itself, because agents follow prompt instructions only some of the time:
-
passwithout code references is rejected. So ispassjustified by "the scanners found nothing": no Semgrep signal doesn't prove there's no bug. - AI never changes a finding's status. The agent files it as
suspected; a human setsverified,false_positive,accepted_riskorfixed. AI checks only add an opinion. - Every verdict is tied to a commit and its evidence files. When those files change the cell goes
stale, and you recheck that cell, not the whole module.
Findings also get an independent check. The original session tends to defend its own conclusion, so the refuting session gets only the claim (title, CWE, module, description, references) and never sees the recommendation, the author or the transcript. Its file access is limited to the working directory; otherwise it could read the first session's history from ~/.claude/projects and the whole point would be gone.
What the agent sees together
This was the reason for the whole project, so concretely:
- Entry point map. HTTP, gRPC, GraphQL, queues, cron, CLI and WebSocket, with handler, middleware, authorization check, parameters and the call chain to dangerous sinks. Versioned by commit.
- Scanner signals. Semgrep, Gitleaks and Trivy run once per commit; results are normalized and offered to the agent as hints through
get_signals. You can import SARIF from CodeQL or a corporate scanner. Suppressions keep a reason, an author and an expiry date. - SCA with reachability. A dependency inventory, a filter without AI over tool reports, and a check of whether the vulnerable code in a package is actually called. A reachable vulnerability becomes a finding; an unreachable one stays listed with an explanation.
- Requirements and history. The agent sees past findings and similar checks, so duplicates merge instead of piling up.
Around the audit there's more. A PoC tester starts a test instance from a worktree and checks a finding with requests only to base_url, inside an OS sandbox. Branch checks compare a pull request's changes against a document set before merge. Schedules run all of it on cron with token limits. Every GUI action is also in the CLI, so you can put aiegis on a server and call it from CI.
Nothing is hardcoded: prompts, models, permissions and limits live in editable YAML and Markdown with a cascade of global → project → role → module → matrix → item. Want Sonnet at high effort for the crypto module and Haiku everywhere else? Two lines in .aiegis/config.yaml, committed and reviewed with the code.
The benchmark
I needed to know whether this works, not whether it looks like it works. So I ran aiegis on CWE-Bench-Java, a set of real CVEs in Java projects from the IRIS project (data under MIT).
Method
I picked 15 CVEs across 14 CWE classes with mechanical filters: linear history, a fix touching at most four files, fix files no more than 5% of the module, Apache, MIT or MPL licenses. Another 21 CVEs went into a holdout set by a deterministic draw, without looking at the code, and nothing was tuned on them.
I worked hard to remove hints. Each version's code went into a fresh repository as a single neutral commit, with no tests, release notes, links or agent settings. If the agent ran into a CVE or GHSA id in the code, the run stopped. There's one requirement per CWE class, phrased in general terms, never "find the bug in XmlParser.java".
A hit is a finding in the fixed file with the right CWE family. Recall is measured on the vulnerable version; false positives at the fix location in the fixed version. needs_human answers and findings without a CWE count as "no answer", not as misses. Intervals come from bootstrapping over CVEs.
Results
Default config: Sonnet at low effort for verdicts, Haiku for triage, a 150k token limit per run, scanners on. The main set ran three times, 90 runs in total.
| Main set (15 CVEs) | Holdout (21 CVEs) | |
|---|---|---|
| Recall (file + CWE family) | 57% [36–79%] | 40% [20–60%] |
| Exact method found | 38% | 15% |
| False positives on the fixed version | 9% [0–24%] | 11% [0–28%] |
| Cost | $0.22 per run | $9.59 for the whole set |
Hits didn't vary across the three repeats for a single CVE: if aiegis finds a vulnerability, it finds it every time. The first full run cost $20, about 52k billable tokens per run on average.
Fix verification did better than I expected. With an empty commit on top of vulnerable code, the agent should say "not closed", and it did in 24 of 24 cases. On the commit with the real fix it said "closed" in 18 of 24. All six misses came from two CVEs where neighbouring defects remained in the same file, so the agent wasn't entirely wrong.
Refutation is weaker: 28 confirmed, zero refuted. Either the original findings were good or the refuting session is too lenient, and this set can't tell which.
Where it misses
Two patterns. In two CVEs triage said n/a because the code was "a library without entry points": no HTTP endpoints, so input rules don't apply, even though the library's public API is the entry point. In three cases the agent found the right class of bug in the neighbouring file.
What prompt changes didn't do
I spent about $40 more on variants: a web and auth focused prompt, a library API prompt, "file every finding", a hint about the empty entry point map, no methodology at all, and Sonnet at medium effort instead of low. None changed the set of CVEs found. They all find the same 8, and the false positive differences stay within one CVE, about 7 points on a set this size.
That turned out to be the main lesson. Prompt polishing won't move recall here; new capabilities will, such as cross-file analysis and smarter handling of libraries with no obvious entry points.
Caveats
The model may have seen these CVEs in training, so absolute numbers are probably inflated; comparing configurations against each other is more reliable than comparing with other tools. 15 CVEs is a small set: one vulnerability is worth 7 points. The benchmark covers only the matrix on ready requirements, not requirement extraction or the entry point map. And it's Java only.
Current state
This is 0.x, early. Data and config formats may still change. macOS is the main platform; Windows and Linux builds are produced automatically and get less testing. Builds aren't signed.
You need Claude Code installed and logged in (claude on PATH), or pi with a provider configured if you want another model. Semgrep, Gitleaks and Trivy are optional; without them those scanners are skipped.
git clone https://github.com/seelentov/aiegis.git cd aiegis npm install npm run setup # rebuilds node-pty for Electron npm start Builds are on the releases page.
Where I need help
I'm building this alone, and my view is limited to the stacks I've tested on. So, specific asks:
Run aiegis on your own project, especially if it isn't Java: Go, Python, Node, PHP. I want to know where triage goes wrong, where the entry point map loses routes and what the audit cost you. A bug report with version, OS and "what I did, what I expected, what happened" already helps a lot.
If you're on Windows or Linux, try the builds. There are almost certainly problems there I can't see.
If you use pi with non-Anthropic models, I'd love to know how aiegis behaves with them. A separate extension enforces role permissions for pi, and it needs real-world testing.
And code: new scanners and normalizers, better review methodology, translations, docs. The one house rule is that nothing is hardcoded; everything goes through defaults/ and the config cascade. See CONTRIBUTING.md, and for a large change open an issue first so we can agree on the approach.
If you have an idea for measuring tools like this more honestly than on CVEs the model may have seen in training, tell me in the comments. I don't have a good answer to that yet.
Repository: github.com/seelentov/aiegis
