Coinbase's 9B Fraud Model Beat Opus 4.5: Copy the Pattern With a Verifiable Reward, Tier-Only Output, and Block-Only Harness

Attributed compile + one small offline run on synthetic data

Primary sources (Coinbase Engineering, "Owning Intelligence" series): Part 1/3 — Build by Yao Ma, Bo Lei, Xuwei Tan (2026-10-06), Part 2/3 — Evaluate by Yao Ma, Xuwei Tan (2026-10-07), and Part 3/3 — Own by Yao Ma, Xuwei Tan, Akshit Trehan, Aman Choudhary with the Anyscale team (Eric Tang, Sumanth Hegde, Seiji Eicher, Jess Kong) (2026-10-08)

Also: Brian Armstrong on X (2026-10-11); secondary coverage by Dealroom, Crypto Briefing (Diego Almada Lopez), CoinCentral, and HDATF AX case 137

Every benchmark, latency and cost number about Coinbase's system is Coinbase's, measured on their private data. I did not train anything. The run below is a tiny eval harness on a synthetic, seeded transaction set.

The headline that went around X this week: Coinbase post-trained open-weight Qwen3.5-9B on its own Onramp fraud data, and it beat Claude Opus 4.5 on Coinbase's fraud benchmark. Brian Armstrong's take was that specialist models will "out perform on more narrow tasks."

The model swap is the least copyable part. The useful part is the system around it. That's what this post covers, plus a harness you can run on a laptop in under a second (the optional small-model reviewer takes a few minutes on CPU).

What Coinbase actually built

From Part 1, in my own words:

1. Layered, not replaced. Onramp sends a transaction to a Risk Service. The existing ML model and rules engine screen it first. The LLM agent sits downstream and reviews only selected transactions, using recent transaction history plus written review guidance and calibration examples.
2. Tier-only output. The model returns a risk level (their example is RISK_LEVEL: 5) with no explanation. With a ~7,000-token prompt, Opus had a P50 of about 1.5 s.
3. The harness decides. Application code builds the context, parses the tier, and applies the decision policy. The model never executes a payment action.
4. Block-only authority. The agent can block something upstream allowed. It cannot override an upstream block.

In a 50% traffic A/B test, the agent-enabled arm had 30% fewer fraudulent transactions and 22% less fraud value. Coinbase estimates fraud savings at roughly 3× the inference cost.

Part 2 built a fixed benchmark: 16,140 transactions from 7,293 users, 813 confirmed fraud, taken from nine weeks before the agent rolled out, so the agent's own blocks couldn't bias which outcomes got observed. Newer Opus, Sonnet and GPT versions each scored lower on recall, F1 and dollar-weighted recall than the version before. GPT-5.6 (sol) gained 11.5 points of precision but lost 20.7 points of recall and 21.8 of dollar-weighted recall.

Part 3 is the post-train: RLVR with GRPO and LoRA on SkyRL (on Anyscale), run on one RTX PRO 6000 96GB for under $100. Against Opus 4.5 as a single-pass decision agent: +7.5 precision, +12.0 recall, +9.6 F1, +35.4 dollar-weighted recall. The figure caption says these are percentage-point differences. Serving uses Ray Serve LLM + vLLM on L40S GPUs (2–8 replicas). P50 was 0.683 s vs 1.515 s, P99 1.036 s vs 3.312 s. Coinbase notes that latency and detection were measured in separate evaluations. They also say the unadapted Qwen base "performed poorly."

Why the reward is "verifiable" here

Most RL-for-LLM setups need a judge model to say whether an answer is good. Fraud doesn't. Historical transactions eventually get an outcome: chargeback or confirmed fraud, or not. So the reward is just a deterministic function of (model output, known label). Nothing gets graded on how persuasive it sounds.

Part 3 adds two details worth copying:

• Format checks. Unparseable output gets penalised.
• Class balance. If fraud is rare, "always say low risk" looks accurate. Their reward weights fraud and legit examples so a constant answer can't farm reward.

GRPO then samples a group of answers to the same prompt and pushes the model toward the ones that scored above the group mean. You don't need a value network or a judge.

Why dollar-weighted recall, not just F1

F1 counts incidents. One missed $5,000 cash-out counts the same as one missed $12 card test. Dollar-weighted recall asks what share of the fraud value you caught. Part 2's point is that a model can miss relatively few transactions and still miss a large share of the money.

The run: a laptop-sized version of the pattern

I wrote fraud_tier_eval.py, about 200 lines of plain Python, plus an optional small-model reviewer. It runs on a synthetic set (seed 18): 600 transactions, 47 labelled fraud worth $68,484.50 in total. I wrote the generator with three fraud shapes: card testing (many small buys), cash-out (one large buy, fresh card, new wallet), and quiet (mid-size, geo mismatch only). I designed the shapes, so the results show how the metrics behave. They don't show which reviewer is good.

The flow mirrors the pattern. Upstream rules block first. Routing picks 579/600 for review. Each reviewer returns only a string like RISK_LEVEL: 4. The harness parses it with a strict regex and blocks at tier ≥ 4:

def decide(upstream, reviewer_text): if upstream == "BLOCK": return "BLOCK", "upstream_block_kept" # reviewer can never release  if reviewer_text is None: return "ALLOW", "not_routed" tier = parse_tier(reviewer_text) # ^RISK_LEVEL: ([1-5])$  if tier is None: return upstream, "parse_error" # log it, keep upstream decision  return ("BLOCK", "reviewer_block") if tier >= 4 else ("ALLOW", "reviewer_allow") 

The reward function is the part a GRPO environment would call:

w_f, w_l = n / (2 * n_fraud), n / (2 * n_legit) def reward(text, is_fraud): tier = parse_tier(text) if tier is None: return -1.0 correct = (tier >= 4) == is_fraud return (1.0 if correct else -1.0) * (w_f if is_fraud else w_l) 

Printed results (2026-10-14, 8-vCPU Xeon, Python 3.13, CPU only):

What the run showed:

• F1 and dollar-weighted recall picked different winners. The velocity reviewer caught 24/29 card tests and none of the cash-outs that upstream let through: $-recall 0.603 vs 0.598 for upstream alone. The context reviewer caught 12/12 cash-outs ($-recall 0.979) but blocked $49,411.22 of legit purchases. Neither metric alone is enough. Track precision and legit-dollars-blocked next to them.
• A generic small model added nothing here. LiquidAI's d1-omni-600M, prompted with a plain-English description of each transaction, added 18 blocks and every one was a false positive. Its recall stayed at the upstream 0.319. Median review time was 193.6 ms. That's one model on toy data, but it fits Coinbase's note that the unadapted base was weak and adaptation did the work.
• The class-balanced reward caught the lazy policy. "Always RISK_LEVEL: 1" had 0.922 plain accuracy and a mean reward of −0.000. The d1 reviewer scored −0.033.
• The guard held, but it limits direction, not damage. A deliberately buggy reviewer emitted RELEASE, RISK_LEVEL: 9 and prose. That caused 344 parse errors, and every upstream block stayed blocked (asserted). It still added 111 random blocks, 104 of them legit. Block-only authority stops wrongful releases. It doesn't stop a broken model from blocking good customers, so alert on block rate too.
• GRPO-style group demo: for one $4,733.38 cash-out, RISK_LEVEL: 5 and RISK_LEVEL: 4 scored +6.383 (advantage +0.935) and RISK_LEVEL: 2 scored −6.383. A correct tier with an explanation appended scored −1.0, so the format contract is part of the reward.

Not done: no fine-tuning, no GRPO training run, no Qwen3.5-9B, no Coinbase data, no latency comparison with any frontier model. The set is tiny and synthetic. Files are in runs/18-coinbase-specialist/ (fraud_tier_eval.py, run_output.txt).

When not to copy this

• No trustworthy labels. RLVR is only as good as the outcome label. If fraud confirmation is slow, noisy or missing, the reward becomes guesswork. Coinbase notes labels arrive late even for them.
• Low volume. Coinbase's benchmark alone had 813 confirmed fraud cases. If you have dozens, a prompted frontier model with a fixed eval set is cheaper to run and to reason about. That's my rule of thumb, not a Coinbase threshold.
• Policy that shifts weekly. Part 1 points out that you can change natural-language guidance without retraining. Baked-in weights can't do that. If the rules move faster than you can retrain and re-evaluate, keep the policy in the prompt and the harness.
• No owner for the lifecycle. Part 3's own checklist is to evaluate on domain outcomes, version weights, prompts and policy together, and keep a tested rollback. If no one owns those, don't self-host a model that sits in a payment flow.

How I'd start this week

1. Freeze a labelled eval set from a period before any model touched decisions.
2. Make the model return one parseable token. Put the policy in code.
3. Score every candidate on precision, recall, F1, dollar-weighted recall and legit-dollars-blocked.
4. Only then consider RL. The reward function you need is already in your eval harness.

YongBo Yu is an AI engineer in Toronto building LLM workflows and agent systems. More at yongbo-yu.vercel.app and GitHub.