"How much does it save?" is an easy question to answer badly. The same plugin, on the same 400-case workload, produced 86.1% fewer billed tokens at one threshold — and 8.5% at another. Both numbers are true, and neither means anything without its conditions.

This post is about the measurement behind them: why "token savings" is hard to pin down inside an agent pipeline, what we count and at which granularity, where each digit comes from, and how to rerun the whole thing on your own traffic.

The setup. dsh-phocinae is a local decision plugin for the DeepSeek Harness agent (dsh): two tools, phocinae_ask and phocinae_gate, backed by a small local model that answers typed decision questions in a single forward pass. The workload for this measurement is the published typed-decisions row set — 400 cases, 2,000 decisions — with the local model (Phocinae-Largha-150M-v1, CPU, fp32) paired against a flash-tier hosted model (deepseek-flash, temperature 0). Hosted-side token counts come from the API's own usage counters; everything else was measured with the scripts under bench/, and the raw results are committed under bench/results/. The run is a snapshot — plugin 0.2.1, dated 2026-10-08. "Savings" throughout means billed hosted tokens avoided.

Why "token savings" is hard to measure

There is no single meter. The hosted provider meters the tokens it bills you. A local decision bills nothing — but it is not free. Its tool definitions ride in every request, and each call's arguments and rendered result enter the transcript, where they get billed like any other token on the next request. "Saved" always means "not sent", and you have to be explicit about what you counted.

The escalation unit is not the decision. When escalation happens, the hosted model is called with a whole case — one state plus its five typed questions — so if any single decision in a case is unsure, the case is re-answered wholesale. Counting "decisions kept local" and counting "cases that never hit the API" are two different measurements that will give you two different percentages.

The threshold is a product decision. The plugin produces an escalate signal, not a savings guarantee. τ — the confidence below which an answer is marked unsure — is chosen by the caller, and it is the difference between the 86.1% row and the 8.5% row below. A savings claim without a τ is half a sentence.

The workload decides. Ours is a machine-readable question set, not a live multi-turn agent session. That is a limitation with a number attached; more on it at the end.

The measurement design: three counters

We count tokens in three places.

1. Per request — the fixed part. While the plugin is loaded, both tool definitions ride in every request's tool list: 473 tokens for phocinae_ask plus 212 for phocinae_gate, or 685 tokens per request. These were not estimated; they were counted with the provider's own tokenizer. And note the scope: that number is the plugin's addition to a request, not the request itself.

The approval gate is the exception. It runs as a pre-execute hook, not as a message, so it adds zero conversation tokens — whatever it costs in wall time, it never enters the transcript.

2. Per decision — the round trip. One local call costs 99 tokens of arguments out and 135 tokens of rendered result back: 234 tokens per decision in steady state. A decision that is the first in its session also carries the fixed 685, for 919 total. The fixed counter amortizes from there; how fast is the next section.

3. Per case — the unit the hosted model is actually called with. Every hosted response reports usage, and across the 400-case run those counters summed to 1,222,211 tokens (290,606 in, 931,605 out — about 76% of the bill is the model writing its answers). That is 3,055.5 tokens per case. The local side of the same run: no billed tokens, by construction. A decision is a forward pass, not a generation — the server's usage field reports zero output tokens.

The results, with the conditions attached

Read every number below as one row of a table whose columns are: workload, model pair, τ, and whether the plugin's own footprint is included. All savings here are counted in billed hosted tokens, at the case level. Accuracy is reported next to savings, because a cheap wrong answer is not a saving.

The pure local path. Every decision answered locally, nothing escalated. Against sending everything to the hosted model, the local path saves 69.9% of tokens at one decision, 90.1% at ten, and 92.3% at a thousand. It saturates near 92%, not 100% — the missing part is exactly the fixed counter: tool definitions and round trips stay in the caller's context no matter how large the session grows.

Hybrid at τ=0.4 — the row we would point you to on this workload. 27 of 400 cases were escalated (6.8%). Billed tokens: 82,499 instead of 1,222,211. Count the plugin's own footprint — 685 per request plus 234 per kept decision — and the saving is 86.1%. Accuracy: 0.7305, against 0.7085 hosted-only and 0.7320 local-only. On this workload, escalation was close to a wash. (Both accuracies come from this protocol's rendering: paired and directly comparable to each other, not the model card's standalone figures.)

Hybrid at τ=0.6 — the honest row, and the plugin's shipped default. 363 of 400 cases were escalated (90.8%). Model-only saving: 9.2%. With the plugin's footprint included: 8.5%. Accuracy also slipped to 0.7115 — below the pure-local 0.7320 on this workload, because escalated cases are re-answered wholesale and the hosted model is weaker on this workload's choice-type questions (0.6793 vs the local model's 0.7529, on 1,400 of them).

Why does 0.6 escalate so much? Because the confidence distribution is concentrated near the middle — the median decision sits at 0.6294 — and one unsure decision is enough to flag its whole case. 851 of the 2,000 decisions (42.6%) sit below 0.6; counted per case, that escalates 90.8% of cases. A threshold that sounds conservative is, on this distribution, nearly an opt-out.

Cost, at the paired model's published unit prices ($0.28 per million input tokens, $1.10 per million output): the 400-case run comes to $1.1061 hosted-only and $0.0747 at τ=0.4. A local-only run has no billed tokens.

One discipline note before the digits drift: two savings figures from the same run can still be incomparable if one counts decisions routed and the other counts tokens billed. Every number here carries its counter, its unit and its threshold. When you see a savings headline without those three labels, you don't yet know what question it answered.

What we did not measure

  • Live agent sessions. 400 machine-readable cases is a bench, not a session trace. Real sessions have long histories, more tools, more turns. The next experiment we owe ourselves: the same task list (30+ tasks) with the plugin on and off, aggregating the API's usage and counting escalations, false blocks and task success alongside tokens. Until that runs, treat the numbers above as a calibrated bench, not a promise.
  • GPU latency. Every latency figure in this post is CPU, fp32, one machine: 42.6 ms for a local decision's HTTP round trip, ~21 ms mean to screen a command through the gate, 0.02 ms for an allow-listed command. A GPU figure also circulates — fp16, p50 21.0 ms on an RTX 5090. That one is the model release's published number, not a measurement of ours; we have not reproduced it on this bench — that measurement is still pending.
  • Portability of τ. 0.4 is not a universal constant; it is what this workload's confidence distribution produced. Re-scan on your traffic.

Reproduce it on your own traffic

Everything above is re-derivable from bench/:

# 1. serve the local decision model python -m phocinae.main # 2. the token-savings run — point it at your rows and your model pair node bench/token-savings.mjs --rows 400 --dataset ./your_rows.jsonl --model <hosted-model> # 3. your plugin's own footprint: tool-definition tokens, round trip, gate wall time node bench/context-cost.mjs # 4. amortization and hybrid rows, plugin overhead included node bench/economics.mjs # 5. the gate alone, against the annotated command set node bench/gate-by-mode.mjs

Sweep τ across {0.3, 0.4, 0.5, 0.6} and read accuracy next to savings: on this workload, pushing τ up bought less saving and worse accuracy. bench/cases.json holds 53 annotated commands — replace it with your own traffic to measure your gate instead of ours.

The measure of a savings claim is whether you can rerun it. Ours: 86.1% at τ=0.4 on 400 cases, plugin overhead included — and the 8.5% row at τ=0.6, reported because a knob you don't report is a knob you don't understand. Point the bench at your logs, and trust your own row over anyone's — including ours.

  • Plugin: github.com/Phocinae/dsh-phocinae — bench/ in the same repo holds the scripts and raw results behind every number above
  • Model (Apache-2.0): huggingface.co/Phocinae/Phocinae-Largha-150M-v1