8 Frameworks in the Demo, 3 Rows in the Conformance Table: I Ran Microsoft's agent-hooks CTK

Eight frameworks in the demo, three rows in the conformance table. I ran the kit to see which number your framework is in.

The agent-hooks launch post makes a specific, checkable claim. Same support scenario — a $840 refund against a $500 cap — run "across eight frameworks: LangGraph, the OpenAI Agents SDK, Microsoft Agent Framework, Semantic Kernel, LlamaIndex, CrewAI, the Claude Agent SDK, and a bare reference host". Every one of them, it says, "produces the identical 20-row decision stream".

That sentence is doing a lot of work. The same repo ships the thing that decides who is supported: a conformance test kit and a claim table. So I installed the SDK and ran the kit locally, then read the table.

Short version: the kit runs, it's fast, and it's honest about what it tested. The gap is between the demo and the claims table, and it's larger than the launch post suggests.

Running the kit

There is no CLI. The CTK is a Python module you call:

from agent_hooks.ctk import load_vectors, run_vectors from agent_hooks.ctk.reference import ReferenceHarness vectors = load_vectors() results = run_vectors(ReferenceHarness, vectors) print(len(results), sum(r.status == "pass" for r in results)) # 73 69 

On agent-hooks-sdk==0.1.0b2, Python 3.13, that's 73 vectors, 69 pass, 4 skip, 0 fail — in 0.37 seconds. No model, no network, no API key. It's the one part of this stack that is trivially reproducible.

The four skips are the interesting part, because they're not failures:

AH-CTK-110 skip incremental mediation: a fully permitted stream is released segment by ... AH-CTK-111 skip incremental mediation: a terminating deny withholds the unreleased rem ... AH-CTK-112 skip incremental mediation: residue no emission evaluated fails closed at e ... AH-CTK-113 skip incremental mediation: durable incorporation is gated by the same disc ... 

The reason field is missing capabilities: ["incremental_output"]. The kit is capability-gated: a host declares what it can do, and vectors outside that declaration are skipped rather than failing. Drop all four and the whole streaming/incremental part of the contract goes untested.

That's defensible design, and the repo is upfront about it. conformance/CLAIMS.md defines "conformant" as passing 100% of the CTK vectors applicable to the host's declared surface, and explicitly requires a host with incremental mediation to declare incremental_output "so the streaming/incremental vectors run against that discipline instead of being skipped".

But read the claim language as a buyer, not as a spec author: "100% of applicable vectors" and "100% of vectors" sound the same in a press release, and they are not the same. The 4 skipped vectors are exactly the ones that answer "what happens to a deny that arrives after I already streamed content to the user". That is the question I would want answered before I put a deny in front of a token stream.

Three rows

Here is the whole claim table, as of this run:

Three rows. Six of the eight frameworks named in the launch post — LangGraph, the OpenAI Agents SDK, Semantic Kernel, LlamaIndex, CrewAI, the Claude Agent SDK — have nothing filed.

I went looking for adapters before I looked for claims, and the Python package confirms it: grep -ril for langgraph, langchain, openai, crewai, llama_index, semantic_kernel, claude across the entire agent_hooks package returns zero hits. The only host binding shipped in Python is create_agent_hooks_middleware, in agent_framework. If you want agent-hooks inside a LangGraph or OpenAI Agents SDK loop today, you write the adapter, because the four-method harness interface is the interface and nobody has written yours yet.

This is not a scandal, and it is still the whole story

Nothing here is dishonest, and I want to be precise about why.

The launch post says we run this same scenario across eight frameworks. That's true — Microsoft wrote a demo suite and wired it to eight hosts, and the interception contract is host-agnostic enough that the guard code travels. What the sentence does not say is that the shipped SDK binds exactly one of them. The demo is Microsoft's code on eight hosts; the conformance claim is a per-host artifact someone has to file in a PR, and the repo says plainly: "If you maintain a framework and want the conformance report with your name on it, the harness interface is four methods, and we'll do the integration work with you."

Fair. Also, that offer is currently unaccepted by six of the eight.

The repo is equally clear about the limits, which I'd rather quote than paraphrase: "A conformance claim is not a security certification... does not test adversarial bypass, does not assure that the production code path matches the harness." The MAF claim is one release behind too — it cites core 1.13.0 at commit 4b1afd9052, while 1.21.0 was current when I installed it, and the claim pins a 51-vector corpus against the 73 the SDK loads now. Claims age.

The tell is in LangGraph's own tracker

If you want a concrete read on how far the "eight frameworks" reach actually goes, don't read the launch post. Read langgraph#8102, opened 2026-06-16, still Open, no assignee, labeled external:

LangGraph routes model tool calls into ToolNode (_run_one / _arun_one) and then executes the selected tool. In practice, there is no simple first-class before_tool_call hook with a stable contract to intercept, inspect, and block/modify a call before execution.

That is a request for the exact seam agent-hooks builds its entire model on. It's four months old and unanswered. A framework where the pre-execution tool seam is still an RFC has not adopted a contract whose fourth interception point is pre_tool_call — whatever the demo showed.

What I'd do with this

Three things, in order:

1. Treat conformance/CLAIMS.md as the supported-host list, not the launch post. It's one table, it names commits, and it's the repo's own arbiter of what "supported" means.
2. Run the CTK yourself before adopting. It's four lines, sub-second, and it tells you which parts of the contract your host actually exercises. If 4 vectors skip, you know which class of question is unanswered.
3. If your framework isn't in the table, the honest read is "the spec is portable, the adapter isn't written". That's a real decision, not a formality — the adapter is the part where the verdict contract meets your loop's actual control flow.

The contract itself looks well-built, and capability-gated conformance with per-part reports is more honest than most governance releases manage. The mistake would be reading "eight frameworks" as a compatibility statement. It's a demo.

Figure 1 — my CTK run and the claim table side by side.

Repro: from agent_hooks.ctk import load_vectors, run_vectors plus the SDK's reference harness. Sub-second, no model, no network. Numbers above: agent-hooks-sdk==0.1.0b2, Python 3.13, run 2026-10-14.