Audit the Numerals: The Part of Generated Docs Human Review Skips

Generated documentation fails where reviewers skim: on the numbers. A reader forgives a clumsy sentence about retry behaviour, then loses an afternoon to a timeout value the page invented. Treat numerals as their own machine-checked class of claim, so a model can keep drafting prose while a short standard-library script fails the build when a number has no fact behind it. The cost is one fact file, roughly seventy lines of Python, and one CI step.

Why the digits survive review

Reviewers of generated docs check structure, coverage, and tone, and digits slip through because they look factual. Two failure classes do most of the damage: invented precision, where the model rounds or estimates a value nobody measured, and unit drift, where the right number arrives attached to the wrong unit. Both read fluently, which is exactly why a human pass over a diff does not catch them. The fix is not a stricter prompt, because prompts reduce leakage without proving absence.

Step 1: turn config into a fact file with source pointers

The audit needs a machine-readable set of numbers it is allowed to see, each pointing back at the code that produces it. Name each key so that its last segment is the unit, and the script can check units without any extra schema.

{ "max_retries": { "value": 3, "source": "src/config/limits.py:12" }, "timeout_seconds": { "value": 30, "source": "src/config/limits.py:14" }, "max_payload_mb": { "value": 2, "source": "src/api/schema.py:8" }, "batch_jobs": { "value": 4, "source": "tools/codegen.toml:6" } } 

Generate this file in the same job that already reads your config, rather than typing it by hand. A hand-maintained fact file becomes a second source of truth and rots within a release cycle. If a value cannot be extracted automatically, it does not belong in the file.

Step 2: write the ledger script

The script blanks out code fences, inline code, links, and URLs while preserving line numbers, then compares every numeral and every word-numeral pair against the fact file. Note the four-backtick fence below: the pattern itself contains triple backticks.

#!/usr/bin/env python3 '''numeral_ledger.py: fail when generated prose states a number no fact backs.''' import json, pathlib, re, sys FENCE = re.compile(r'```.*?```', re.S) INLINE_CODE = re.compile(r'`[^`]*`') LINK = re.compile(r'\[[^\]]*\]\([^)]*\)') URL = re.compile(r'https?://\S+') WORDS = {'two': 2, 'three': 3, 'four': 4, 'five': 5, 'six': 6, 'seven': 7, 'eight': 8, 'nine': 9, 'ten': 10, 'twelve': 12, 'twice': 2, 'thrice': 3} VAGUE = ('several', 'few', 'many', 'most', 'some', 'couple', 'handful', 'numerous') UNITS = { 's': 'second', 'sec': 'second', 'secs': 'second', 'seconds': 'second', 'second': 'second', 'ms': 'millisecond', 'milliseconds': 'millisecond', 'kb': 'kilobyte', 'kilobytes': 'kilobyte', 'mb': 'megabyte', 'megabytes': 'megabyte', 'retries': 'retry', 'retry': 'retry', 'attempts': 'attempt', 'tries': 'attempt', '%': 'percent', 'percent': 'percent', 'jobs': 'job', 'workers': 'worker', } NUMERAL = re.compile(r'(?<![\w.])(\d[\d,]*(?:\.\d+)?)\s*(%|[A-Za-z]{1,12})?') WORD_PAIR = re.compile( r'\b(' + '|'.join(WORDS) + r')\s+(' + '|'.join(sorted(UNITS)) + r')\b', re.I ) VAGUE_RE = re.compile(r'\b(' + '|'.join(VAGUE) + r')\b', re.I) def blank(text, pattern): # Keep the newline count so reported line numbers match the source file.  return pattern.sub(lambda m: '\n' * m.group(0).count('\n'), text) def load_facts(path): raw = json.loads(pathlib.Path(path).read_text()) values, unit_index = set(), {} for key, rec in raw.items(): value = float(rec['value']) values.add(value) tail = UNITS.get(key.rsplit('_', 1)[-1].lower()) if tail: unit_index[(value, tail)] = key return values, unit_index def audit(md_path, facts_path, allow): values, unit_index = load_facts(facts_path) prose = pathlib.Path(md_path).read_text() for pattern in (FENCE, INLINE_CODE, LINK, URL): prose = blank(prose, pattern) blocking, advisory = [], [] for lineno, line in enumerate(prose.splitlines(), 1): if any(a.search(line) for a in allow): continue for match in NUMERAL.finditer(line): raw, unit = match.group(1), (match.group(2) or '').lower() value = float(raw.replace(',', '')) canon = UNITS.get(unit) if value not in values: blocking.append((lineno, f'unbacked numeral {raw!r}')) elif canon and (value, canon) not in unit_index: blocking.append((lineno, f'unit drift: {raw} {unit}')) for match in WORD_PAIR.finditer(line): value, unit = WORDS[match.group(1).lower()], UNITS[match.group(2).lower()] if (value, unit) not in unit_index: blocking.append((lineno, f'unbacked word numeral {match.group(0)!r}')) for match in VAGUE_RE.finditer(line): advisory.append((lineno, f'vague quantifier {match.group(0)!r}')) return blocking, advisory def main(argv): strict = '--strict' in argv args = [a for a in argv if not a.startswith('--')] md_path, facts_path = args[1], args[2] allow = [re.compile(r'\bv?\d+\.\d+(\.\d+)?\b')] # version strings  blocking, advisory = audit(md_path, facts_path, allow) for lineno, msg in blocking + advisory: print(f'{md_path}:{lineno}: {msg}') print(f'{len(blocking) + len(advisory)} finding(s), {len(blocking)} blocking') return 1 if (blocking and strict) else 0 if __name__ == '__main__': sys.exit(main(sys.argv)) 

Step 3: run it on a fixture and read the output

Start with a page you already published, so the first run measures real leakage instead of a synthetic example. The fixture below mixes a passing numeral, a failing one, a numeral hidden in inline code, and two vague quantifiers.

The client makes three retries before it gives up, and each attempt waits 45 seconds. After a failed upload the SDK drops the batch and returns an error. A 2 MB payload is rejected outright, so keep batches small. Regenerate the client with `python -m codegen --jobs 3` when the schema changes. In practice, most teams need a couple of runs before the output stabilises. 

python3 numeral_ledger.py docs/payload.md facts.json --strict echo "exit=$?" 

docs/payload.md:1: unbacked numeral '45' docs/payload.md:8: vague quantifier 'most' docs/payload.md:8: vague quantifier 'couple' 3 finding(s), 1 blocking exit=1 

The run passes three retries and 2 MB because both match a fact key with the expected unit, and ignores 3 because it sits inside inline code. Line 1 fails because 45 seconds exists nowhere in the fact file, which is usually the moment someone discovers the draft contradicted the real timeout. Run without --strict for two weeks and compare counts before you make it blocking.

Step 4: draft first, audit second, never the reverse

Put a short contract in front of the model so the audit has less to catch, then keep the audit regardless of how well the prompt performs.

Draft one docs page. Use only the FACTS block for any number, duration, size, or count. FACTS entries look like: key = value (unit). Never invent a unit. Never round. Never write 'about' or 'roughly' before a fact. If a sentence needs a number that is not in FACTS, write TODO(fact) and continue. 

Free model access changes the economics here, because regenerating a page on every config change is only worth doing if the audit is cheap enough to run every time. If you draft through MonkeyCode's free model access and run short jobs on the free server option, keep the ledger as a separate job from the drafting call. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The separation matters because a failing audit should never silently cancel the next draft, and a passing audit should never be treated as approval of the surrounding claim.

What the model may draft, and what a human must own

The table is the part that survives tooling changes. Ownership is decided per element class, not per sentence, so a reviewer never has to guess whether a given paragraph was meant to be checked.

Limitations and who should skip this

The ledger proves that a number exists in the fact file; it cannot prove the sentence used it correctly, so a backed value attached to the wrong subject still passes. Ranges and inequalities such as 1-3 retries or under 30s are not parsed and will show up as unbacked until you extend the patterns. Unit conversion is absent, so 1000 ms and 1 s are different facts, and the per-line allow-list for version strings is coarse enough that you should keep version numbers on their own line. Word numerals are recognised only in English and only when directly followed by a unit token.

Avoid this workflow if your docs contain mostly conceptual prose with no extractable config behind the numbers, since a hand-typed fact file is worse than no fact file. Skip it too if your pages change faster than the underlying code, because the audit will be correct and useless at the same time. Teams without an automated config reader should build that extraction step first and revisit this afterwards.

Run the ledger against one published page before you change any prompt, and let the finding count decide whether the workflow is worth the CI minute.