Your Extraction Agent Is 94% Accurate and Still Ships Wrong Quantities

Document-level accuracy hides the errors that cost money. Score fields, weight them by consequence, and gate every prompt change in CI.

A team demos an agent that reads supplier quotes and customer RFQs and turns them into structured line items. The PDF-to-JSON accuracy is reported as 94%. Two weeks into the pilot, a customer receives a quote for 120 units that should have been 12. The description field was perfect. The quantity was off by a digit, and no dashboard showed it, because the metric averaged everything together.

This is the most common gap we see in LLM extraction projects: the model is good enough, and the measurement is not. If you cannot say how often the quantity field is wrong on scanned PDFs from your top five suppliers, you do not know whether the system works.

Averaged accuracy rewards the wrong behavior

Extraction outputs have fields of very different weight. A mangled product description gets noticed by a human in seconds. A wrong unit (m2 versus m3, piece versus pack of ten) or a shifted decimal in a quantity flows straight into a quote, an order, or an ERP posting.

A single accuracy number blends these together. Say a document has 8 fields per line, and the model nails 7 of them on every line. You report 87.5% field accuracy, and the one wrong field per line is the quantity. The number looks fine. The system is useless.

Document format makes it worse. Native PDFs, scanned faxes, Excel sheets with merged cells and hand-edited emails behave like different models. An aggregate over all of them tells you nothing about the scans, which is where the failures cluster. In the past month, at least four DACH-funded startups announced agents for legacy-heavy workflows: procurement, B2B trade, travel and building materials (per our market tracking, sourced from Tech.eu weekly recaps). Their domains differ and the documents are messy in the same way.

What we run on our own pipeline

Our internal sales pipeline uses LLM agents to research companies, score leads and draft outreach. None of that output is trusted on its own. Each agent run is checked against acceptance criteria: required fields present, a justification attached to every score dimension, a source URL on every claim. Before any first email leaves, a person approves it. The model drafts; deterministic code and a human decide what ships.

That setup is small, and it taught us the lesson that applies to client work: validation and review catch more than prompt tweaking does. When a field fails a check, the cheapest fix is almost never a longer prompt. It is a rule, a lookup table or a review step.

Build the eval harness before the second prompt revision

The harness is not complicated, and it should exist before anyone rewrites the prompt for the first time. We build these with the client's domain expert, because only they can label the gold set correctly.

Start with a gold set of 50 to 100 real documents, stratified by source: native PDF, scan, spreadsheet, email body. A domain expert labels every line item. Fifty documents with 20 lines each is already 1,000 labeled lines, enough to see patterns. Synthetic documents are not a substitute, because the failures come from real-world mess.

Score each field with its own match rule. Use exact match on article numbers and quantities, canonicalized match on units, fuzzy match on descriptions. Lines must be aligned first, by article number where it exists and by position otherwise. Alignment bugs in the harness produce phantom regressions, so test the aligner on its own.

type Field = 'article_no' | 'quantity' | 'unit' | 'description'; const CRITICAL: Field[] = ['article_no', 'quantity', 'unit']; function scoreLine(pred: Line, gold: Line): Record<Field, boolean> { return { article_no: norm(pred.article_no) === norm(gold.article_no), quantity: pred.quantity === gold.quantity, // Decimal, never float unit: canonicalUnit(pred.unit) === canonicalUnit(gold.unit), description: similarity(pred.description, gold.description) >= 0.85, }; } // CI gate: fail the build if any critical field regresses on any stratum for (const stratum of strata) { for (const f of CRITICAL) { if (recall(results, stratum, f) < baseline[stratum][f] - 0.01) { console.error(`regression: ${stratum}/${f}`); process.exit(1); } } } 

Run the gate on each document type separately. A prompt change that lifts native PDFs by two points and drops scans by six should fail, and an average would let it through.

Pin model versions and record every result. Store each eval run with the prompt hash, model identifier and per-field numbers. When a provider updates a model behind an alias, you will see the shift in the next scheduled run instead of in a customer complaint. Treat prompts as versioned artifacts with the same discipline as code; we wrote about that separately in our post on prompt versioning.

Route to humans with deterministic checks, not model confidence

Self-reported confidence from an LLM is poorly calibrated. A model can return 0.97 on a field it got wrong. Use signals you can verify instead:

• Schema validation with a strict parser (Zod in TypeScript, Pydantic in Python). Reject, do not coerce, anything outside the enum of known units.
• Cross-field arithmetic. If quantity times unit price does not equal the line total within rounding, flag the line.
• Two-pass disagreement. Extract twice with different prompts or models and compare critical fields. Disagreement goes to review. This roughly doubles token cost on the extraction step, so apply it to critical fields only if cost matters.
• Plausibility against history. A quantity of 120 where the same customer has ordered 10 to 15 for two years deserves a second look.

Flagged lines go into a review queue in Postgres with the document image next to the extracted values. Every correction a reviewer makes is stored and, after a spot check, promoted into the gold set. After a few months the gold set reflects your real traffic, and the harness gets sharper without extra labeling projects.

Staffing this

The work splits into three parts: the extraction service (Python or Node.js), the eval and review tooling (TypeScript for the reviewer UI, Postgres underneath), and the labeling process with a domain expert. The first two are ordinary backend engineering. Teams with strong domain knowledge and few backend engineers often skip the harness because it looks like overhead, then spend the pilot debugging by anecdote. A backend engineer who has built one before can stand up a first version quickly, and that costs far less than one wrong quote reaching a customer.

Key Takeaways

• Report accuracy per field and per document type. A blended number hides the failures that matter.
• Weight fields by consequence. Quantities, units and identifiers get exact-match rules and a CI gate; descriptions get fuzzy matching.
• Gate every prompt, model and parser change against a gold set of 50 to 100 real documents, and pin model versions.
• Route to human review using schema checks, arithmetic and cross-pass disagreement. Do not rely on the model's own confidence.
• Feed reviewer corrections back into the gold set so the harness reflects real traffic.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog