We Started With Four Open Agent CLI Questions. We Ended With Two Real Wins.
From the agentpolicy v0.1.0 release: a local-first, Cedar-based policy decision point for AI agents (decide allow, deny, ask, rate-limit, or redact for every agent action, with shipped rule packs). Now on PyPI. This story is the Tier-1 part of the v0.1.0 field test report.
AI disclosure: this article was drafted and edited with AI assistance. The field test, the defects, the fixes, and every number are real and linked below.
At one point, the entire Tier-1 CLI section of the field test was still unresolved.
That was the honest state. Claude Code had not run live yet. Codex had not run live yet. Cursor and Gemini had not been closed out either. If I had stopped there, the report would have been accurate but incomplete.
So I kept going case by case, with one rule: do not claim a harness until it runs with evidence on this box.
That persistence changed the release in a meaningful way. Claude Code and Codex both went live. Cursor and Gemini both got real investigations and honest noes. We shipped the two we could prove and left the other two out.
The starting point
The Tier-1 set in the plan was four CLIs: Claude Code, Cursor, Codex, and Gemini. Early in the sweep, none of them belonged in a green release note. The report had to say that plainly.
This is where the field-test discipline mattered. A case was only real if it ran through the actual harness and left evidence behind: verdict, assertions, transcripts, audit trail, and captured LLM traffic. If it did not, it stayed unproven.
That kept me from turning "probably works" into "supported."
Claude Code: from plausible to proven
Claude Code was the first real win, and it took more than a config toggle. The working harness is in scripts/fieldtest/t1_claude.sh, with the hook entrypoint in t1_evaluate.py and the shared policy in t1_policy.yaml.
Three things had to be figured out:
- The server had to speak Anthropic's wire format. A probe showed the local server could answer
/v1/messageswith a realmsg_*body, so the CLI could talk to it directly. - The hook had to actually fire. My first run looked believable but the audit log was empty. The fix was passing
--settingsexplicitly because cwd settings files do not load headless the way I assumed. - The traffic had to be captured outside the Python process. Claude runs as a separate node process, so the in-process httpx hook could not see its traffic. That is why
t1_proxy.pyexists.
Once that wiring was real, the case passed live: a read executed, a prod write escalated without creating the side-effect file, a destructive command was denied, and the audit trail recorded all three decisions. Six request/response pairs were captured.
Codex: a different surface, still a real win
Codex needed a different investigation. Its harness lives in scripts/fieldtest/t1_codex.sh.
The key findings were different from Claude's:
- Codex no longer supports chat completions. It wants the OpenAI Responses API, and the local server happened to serve
/v1/responsesnatively. -
codex execreads stdin. Left alone in a harness, it quietly waits forever. Closing stdin with</dev/null>fixed the stall. - The provider
base_urlneeded the/v1suffix or every call 404ed.
Codex also forced a more honest test shape. Headless Codex does not expose a pausable per-call approval hook like Claude. So the case proves what the surface actually supports: allow executes, deny holds and replans, plus pre-execution parity against the same in-process policy engine. That is a real pass, just not the same kind of pass as Claude.
The result was still strong: live run, captured traffic, parity audit trail, destructive command blocked, target directory intact.
Gemini and Cursor: two useful noes
Gemini was not a hand-wave. The local install worked. gemini gemma setup brought up a local server. But the CLI's main loop still could not run locally because the local gemma route is classifier-only and the CLI hits an OAuth tier gate before any model routing. That is all documented in the report and backed by the bundle-source read plus the parity stand-in that did run.
Cursor was a different kind of limitation. There is no supported headless surface on this box for the permissions.json flow. Cursor is a GUI IDE, and without a reproducible, evidence-capturing path, there was no honest live case to claim.
Those are not failures to hide. They are useful boundaries. They tell the release exactly what it supports today and what would need to change for the other two to re-enter scope.
What worked
- Persistence after the first honest answer. "Not proven yet" turned into two real live harnesses because I kept going case by case.
- Different proof shapes for different surfaces. Claude and Codex are both real wins, even though one proves a hook path and the other proves a sandbox path.
- Scope honesty. The shipped Tier-1 set is exactly what ran live: Claude Code and Codex.
What did not work
- Treating the four CLIs as if they should all submit to one test shape. They do not.
- Assuming a local install equals a local main loop. Gemini proved otherwise.
- Assuming a popular tool must have a headless testable surface. Cursor did not.
The encouraging part of this story is not "we only got two." It is that we started with four question marks and finished with two live integrations, two investigated boundaries, and a release note I can defend line by line.
More detail and evidence
- The full Tier-1 story: v0.1.0 field test report, especially journal Groups 20, 21, and 22 plus Field Notes 11 through 13.
- The harness files:
t1_claude.sh·t1_codex.sh·t1_evaluate.py·t1_proxy.py·t1_policy.yaml. - The plan and final shipped scope: FIELD_TEST_PLAN.md and the master table note in the report.
- The product: agentpolicy,
pip install agentsec-agentpolicy, PyPI 0.1.0, docs reference, part of the agentic-suite agent-security stack.
Have you had a release where the strongest part of the story was not the things that passed, but the things you proved carefully enough to leave out?

