AI Built My App. Another AI Found the Security Flaw

Two users send the same training-session id to the API at nearly the same time, each into their own account. Both requests come back 204 No Content. One user's data ends up in the other user's account.

That bug was in an app I built with Claude Code, with every security rule I know written into its instructions. An AI pentester spotted the race condition, and I confirmed it. The strangest part wasn't the bug: the agent tried to prove it, and its own model refused to run the test.

I gave Strix, an open-source AI pentester, the source code and a running copy of the API and asked it to break in. This post covers:

1. How I set up an AI pentester against a local copy of my app, on a ChatGPT subscription instead of a metered API key.
2. What the agents tried, what failed, and the one lead that became a confirmed bug.
3. How I triaged every lead by hand, fixed the bug and proved the fix, and why the report you need to read is not the summary.

The target: an app with security built in

The app is for runners: GPS-recorded runs, an interval timer with audio cues, a training plan and goals. The online account keeps all of that synced across devices, along with personal data such as how each workout felt and whether something hurt. That makes broken access control the most valuable class of bug to look for: one user reading or changing another user's runs, plans or account.

The defenses it already had before the scan:

None of that came from a one-off prompt. The repository keeps an architecture decision record (ADR) for every security choice and a set of instructions that Claude Code reads on every task: hash passwords with Argon2id, validate every input with a schema, check ownership on every request. The AI wrote code against written rules, and I reviewed it against the same rules.

I had already attacked it myself. The repository has end-to-end attack scripts that run against the real Docker image: user enumeration, 2FA bypass, mass assignment, a tampered JWT, a forged pagination cursor, IDOR on delete, refresh-token replay, cross-account access. Before the scan I ran all of them against the same image the AI would attack, and every check passed. So the question for the AI was not "can you find the basics?" but "can you find what I didn't think to test?"

There is one more twist. Most of the code was written with Claude Code, the attacker ran on an OpenAI model, and I triaged the results with Claude Code again. One vendor's AI wrote and checked the code, the other's attacked it.

The attacker: what Strix is

Strix runs autonomous AI agents as pentesters. A root agent plans the test and spawns specialized sub-agents, each with its own skills (IDOR, JWT attacks, SQL injection, race conditions). They work inside a Docker sandbox that has a terminal, an HTTP proxy and a browser, so they can read the code, send real requests and try to prove each issue.

That last part is the difference from the scanners most teams already run. A SAST tool matches patterns in code; a DAST tool sends payloads to a URL. Strix's README promises findings "validated through actual proofs-of-concept", which means an agent that reads the code, forms a hypothesis and then sends the requests to confirm or reject it.

In my run, the agents combined classic tools with reasoning. The source-mapping agent ran Opengrep (425 rules over 277 files), a structural search over 8,061 call sites, gitleaks for hardcoded secrets and Trivy for dependency advisories. The live agents then sent requests through the proxy to check what the code suggested.

The setup: model, scope and the flags that matter

strix auth login chatgpt # OAuth sign-in, the same flow as the Codex CLI export STRIX_LLM="chatgpt/gpt-6.1-sol" # chatgpt/<model> runs on the subscription strix -n \ -t ./my-app \ -t http://host.docker.internal:3000 \ --instruction-file ./scope.md \ -m quick --scope-mode full 

The scope file tells the agents what they may touch: the source tree (read only), the one live host, two accounts (alice and bob, each with their own data) and the priorities: first broken access control between the two accounts, then authentication and sessions, then input handling and business logic. Third-party integrations, e-mail and denial of service were out of scope.

Four details worth knowing:

• Set --scope-mode full for a whole-app test. The default, auto, switches to reviewing only the pull request diff in CI runs. My laptop run would not have triggered it, but full makes the scope explicit wherever the command runs, including the day you move it into a pipeline.
• -r is resume, not run name. One of my first attempts died instantly with "Cannot combine --resume with --target".
• The model in the docs was not on my plan. chatgpt/gpt-5.4, the example in Strix's README, came back with "The 'gpt-5.4' model is not supported when using Codex with a ChatGPT account". The model my Codex CLI uses by default worked.
• Local directories are mounted writable. The CLI help says so. I scanned a git archive copy with a neutral folder name, and afterwards I compared it with the commit: the agents had not changed a file.

Using a ChatGPT subscription outside OpenAI's own products is not officially supported by OpenAI, and Strix's source says so in the module that implements it. I accepted that for a one-off test and signed out afterwards. For anything regular, use an API key and the provider's security-testing program: Strix's provider docs say that frontier OpenAI, Anthropic and Gemini models need a cyber access program for security testing. I skipped that line. It came back to bite me, as you'll see below.

The attack, step by step

From the coverage report Strix writes next to the findings: every surface an agent examined, what it tried and the outcome.

Reconnaissance: what does the API look like?

A live agent signed in with both accounts, recorded the token format (a short-lived bearer JWT plus an opaque refresh token) and probed the public surface. It noticed in the source that API docs exist and asked for /docs and /docs/json: both returned 404, because docs are only served when a flag is on. In parallel, the source agent ran the scanners above and built a map of routes to use cases. Gitleaks found nothing; the only key-like strings were test fixtures.

Forging tokens: ruled out

The authentication agent read the token signer, saw the pinned algorithm and the required claims, and then tried to break it anyway: an unsigned alg: none token, edited sub, purpose and aud claims, an empty signature and a malformed bearer header. Every one returned 401. So did an access token used where a pending-2FA token was expected.

The near miss: logging out someone else

The agent controlled both test accounts, so it could play out a leaked-token scenario: what if alice gets hold of bob's refresh token, from a log or a screenshot, and uses it to end his session? Signed in as alice, it called logout with bob's refresh token. The API answered 204 No Content, which looks like success. The agent didn't stop at the status code. It immediately refreshed bob's session, which still worked: the logout use case only revokes a token that belongs to the caller and returns the same response either way. Ruled out, with evidence, in two requests.

Reading someone else's training data: ruled out

Then the attack I worried about most. Each account has its own space for plans and goals, identified by an id in the URL. The agent swapped alice's space id for bob's, and bob's for alice's, on the goal profile and plan preview endpoints. Every foreign id returned 404; the same calls on each user's own space returned 200. It also injected ownership fields into the request body, and the strict schemas rejected them with 400.

Re-authentication, refresh races and 2FA: ruled out

It sent extra userId fields to password change and account deletion (400) and wrong passwords (401). It deliberately did not change real passwords or delete accounts, which is what I'd want from a human tester too. It refreshed one token twice in parallel to look for a race in token rotation: one 200, one 401. And it read the 2FA enrollment code, then checked the edges it could reach without a valid code: an access token used to verify a sign-in, and confirming an enrollment that was never started, both 401.

The lead: two users, one id

While mapping the code, the source agent wrote a note about the repository that saves training-plan sessions:

Each transaction checks [ownership] via SELECT then upserts globally unique resource ID without [space] setWhere in conflict UPDATE. Sequential foreign IDs are rejected, but concurrent creation under two owned [spaces] may race; mapping agent did not dynamically test.

That is a time-of-check to time-of-use race, and it is exactly the kind of bug my own scripts could not find: they send one request at a time. The root agent spawned a dedicated validator to prove it with concurrent requests.

The model said no

The validator never got to its proof. Its call to the model came back with an error from OpenAI's side:

openai.APIError: This content was flagged for possible cybersecurity risk. If this seems wrong, try rephrasing your request. If you're doing authorized security work that requires more cyber permissive safeguards, apply for Daybreak access [...] 

The agent crashed. Later the same filter refused the root agent itself, and the run ended with status failed. The model had been fine reading code, forging JWTs and swapping account ids. In my run, the line was a race-condition exploit. That is one observation, not a rule: it doesn't show that the provider blocks every race-condition test.

This is the risk that's easy to miss in AI pentesting demos: the model's provider decides which attacks your pentester may run, and it can block any request in the middle of a scan. OpenAI offers Daybreak, its trusted access program for cyber work, which loosens these safeguards for vetted defenders (OpenAI says it doesn't remove every refusal). Anthropic has its Cyber Verification Program and Google its Fairwind Program. Without one, an OpenAI, Anthropic or Gemini model can refuse offensive tasks at any point.

What it found

Triage of the leads

A lead is not a finding. Strix filed zero vulnerabilities; it left three leads marked needs_follow_up. I reproduced each one by hand against the local API before calling it anything.

F-01: concurrent writes with the same id cross into another user's data

The pattern, simplified:

// Before (simplified; runs inside one transaction `tx`). async saveSessions(spaceId: string, sessions: Session[]) { if (await this.anyIdOwnedByAnotherSpace(spaceId, sessions)) return false; // SELECT for (const session of sessions) { await tx .insert(plannedSessions) .values({ id: session.id, spaceId, ...values(session) }) .onConflictDoUpdate({ target: plannedSessions.id, set: values(session) }); // no space check } return true; } 

The ids are UUIDs generated on the phone, so the server can't trust them to be new. The SELECT handles the sequential case: a foreign id is rejected with 409, and my own attack scripts confirm it. The concurrent case is different. Each request runs the check and the write inside its own transaction. Under PostgreSQL's default READ COMMITTED isolation, both transactions run the SELECT, both see no row, and both insert. The second insert waits on the unique index, and when the first commits, ON CONFLICT DO UPDATE updates the row that now exists, whoever owns it.

I sent 200 pairs of simultaneous requests, alice and bob each writing a new session with the same id into their own space:

Afterwards the database held 58 sessions whose space did not match their athlete: 48 stayed in alice's space with bob's content, 10 the other way round. The first writer's session disappears from their week, and the second writer's data lives in someone else's account. Both users were told it worked.

Two things keep the severity low. My test gave both users the same id on purpose: it proves the isolation breaks under that condition, not that an outside attacker has a way to learn the id. A real attacker has to know the victim's id before the victim's request commits, and those ids are random UUIDs generated on the victim's phone; nothing in this test showed a way for another user to read them. And the rate limiter cut the attack short: 141 of 200 pairs never reached the database. But "low" is not "no": it is a broken isolation guarantee that my own tests could not see.

The other repositories (recorded runs, completed workouts, goals) already did this right: their upserts carry the owner inside the statement. Only the plan repository had the check in a separate query. Same rules, same AI, one file that drifted.

Fixing it and proving the fix

The fix is to stop asking first. The ownership condition goes into the write, so PostgreSQL decides it under the same lock that detects the conflict:

// After: the upsert only updates a row of the same space. const written = await tx .insert(plannedSessions) .values({ id: session.id, spaceId, ...values(session) }) .onConflictDoUpdate({ target: plannedSessions.id, set: values(session), setWhere: eq(plannedSessions.spaceId, spaceId), }) .returning({ id: plannedSessions.id }); if (written.length === 0) throw new IdOwnedElsewhere(); // rolls back; the API answers 409 

When the existing row belongs to another space, setWhere is false, nothing is updated and returning gives an empty list. That rolls back the whole batch, and the API answers 409 as before. The separate SELECT is gone. Drizzle's setWhere maps to PostgreSQL's ON CONFLICT ... DO UPDATE ... WHERE.

The regression test is a new attack check in the same end-to-end suite: ten pairs of concurrent writes from two spaces, against the real Docker image. Each pair shares one id, and every pair gets a new one, so no pair can interfere with another. It must see one 204 and one 409 in every pair, and no session with an athlete from another space. Each run started from a fresh database, and no pair hit the rate limit. I ran it on the old code first, to be sure it fails for the right reason:

The decision went into a new ADR: an ownership check is part of the write, never a separate read before it. Every AI-written change from now on is checked against it.

I did not re-scan with Strix, because the same model would hit the same filter. The claim is narrow on purpose: the reproduced attack no longer works under the same conditions. Not "the app is secure".

Read the coverage report, not the summary

The terminal summary counts filed vulnerabilities. Mine said zero. The files next to it tell you what that zero is worth:

Strix documents this. Its agent guide says a 0 exit code only covers what was analyzed and tells you to check run.json before calling a run clean. The leads are where the value is: the confirmed bug in this test started as a follow-up note, never as a filed vulnerability.

Methodology and limitations

• The scan was partial. The model's safety filter ended it before the agents for business logic, input handling (SQL injection, SSRF) and cross-space access on the remaining endpoints recorded any coverage. Those areas are untested by this run, not passed.
• One run is one sample. LLM agents are not deterministic, so another run can follow different paths and find different things.
• Scope: the API and its source, in local Docker with test data. Not tested: any deployed environment, third-party integrations, e-mail flows (sign-up and password reset were off because the test stack has no e-mail delivery) and the mobile client.
• Validation: a finding counts as confirmed only when I reproduced it by hand against the local API.
• Cost and time: about 18 minutes of agent time, 201 model requests, 15.6M input tokens (3.8M from cache) and 41K output tokens, and no separate API bill. That came from an unofficial subscription-based integration, not a supported or guaranteed free way to run pentests.
• No human benchmark: this was not compared with a manual pentest, so it says nothing about replacing one.

The verdict

The measurable part: 12 surfaces reviewed, zero filed vulnerabilities, three leads, one confirmed bug of low severity, two leads ruled out by hand, every attack the agents ran on authentication and tokens rejected, and every sequential cross-account attempt rejected; the one that got through was concurrent. The bug is fixed with a regression test that fails on the old code.

My opinion, clearly as opinion: the agents were good at the thing I can't do in a test suite, which is reading the code and asking "what if two of these happen at once?". They were also honest about what they hadn't proven. The weak points were not the agents but around them: a model that can refuse mid-scan, and a summary that says zero when the coverage report says incomplete.

As for the app: the AI appeared to follow the security rules I wrote. The flaw lived in the guarantee I hadn't explicitly required. Every attack that ran against a written rule failed, except one. The rule said "check ownership on every request"; it didn't say where. The check sat in a separate read, and concurrency slipped between the read and the write. Now the rule says "inside the write". Whether the rules hold for SQL injection, SSRF, CSRF and business logic, this run can't say.

Takeaway

• Written rules make AI-written code testable. The ADRs and instructions told me what each defense was supposed to do, so every failed attack was a check against a rule, not a guess.
• An agent reading the code can spot what one-request-at-a-time tests can't. It flagged the race; proving it took two users and concurrency.
• The model's provider is part of your scope. Without a cyber access program, a frontier model can refuse in the middle of a scan.
• Zero findings is not a clean bill of health. Check run.json and the coverage gaps before you trust the summary.
• Triage by hand, then prove the fix. One lead was real, two were not, and the regression test had to fail before the fix to mean anything.

Do your API tests ever send two requests at the same time? I'd like to know how you test races in the comments.

I'm Diogo Soares, a Staff AI Engineer working on AI agents and legacy modernization. If this was useful, follow me here on DEV.to and on Instagram at @diogosoares.dev, where every article becomes a Reel of under a minute. A video walkthrough of this test is coming to my YouTube channel, Diogo Soares Dev, so subscribe there to catch it. Let's also connect on LinkedIn, or check out diogosoares.dev.