Most teams decide which documentation a model may rewrite by gut feeling, and the gut feeling is usually wrong in one specific direction. Models are good at prose and quietly bad at preserving the boring facts: default values, error codes, environment variable names, and status numbers. This article describes a small measurement harness you can run first, so the ownership decision for each doc section comes from a number instead of a vibe.
The workflow is deliberately unglamorous. You freeze a fact sheet from source, strip a section down to its skeleton, let a model redraft it several times, then diff the claims. Sections that keep their facts get a longer leash; sections that bleed facts stay human-owned.
The failure this catches
Human reviewers read generated documentation for flow, and flow is exactly what models are best at. A redraft that removes the default timeout value but reads beautifully will pass most review passes, because the missing token leaves no gap in the sentence. The defect is invisible on the page and only visible against a baseline set.
That asymmetry is why prose review alone cannot gate this work. You need an artifact that lists what must survive, and a mechanical diff that tells you whether it did.
Step 1: Freeze a fact sheet from source
The baseline must come from code, not from the old prose, because the old prose may already be stale. The script below walks Python source with ast and collects signatures, defaults, and identifier-shaped tokens. It is an unexecuted example: run it against your own repository and check the output before trusting any score it produces.
# fact_sheet.py — example, not run against your codebase import ast, json, pathlib, re TOKEN_PATTERNS = { "error_code": re.compile(r"\bE_[A-Z0-9_]{2,}\b"), "http_status": re.compile(r"\b[45]\d\d\b"), "env_var": re.compile(r"\b[A-Z][A-Z0-9_]{3,}\b"), } def code_facts(py_path): tree = ast.parse(pathlib.Path(py_path).read_text(encoding="utf-8")) facts = set() for node in ast.walk(tree): if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): facts.add(f"fn:{node.name}") for default in node.args.defaults: facts.add(f"default:{ast.unparse(default)}") return facts def prose_facts(text): found = set() for label, pattern in TOKEN_PATTERNS.items(): for hit in pattern.findall(text): found.add(f"{label}:{hit}") return found Keep the extractor narrow on purpose. Every pattern you add is a claim you are promising to police, and a noisy extractor will bury the real drops under false alarms.
Step 2: Cut the section down to a skeleton
Split each doc page at heading boundaries and keep the argument structure while removing the sentences. What remains is a numbered outline plus the fact sheet attached to that heading, which is all the model needs and all you want it to see.
Do this in a script rather than by hand, because the skeleton must be byte-identical across the three or more redraft runs. If the input drifts between runs, the score measures your editing, not the model.
Step 3: Run repeated redrafts
Three runs per section is enough to see instability, and forty sections times three runs is one hundred and twenty calls. Point any OpenAI-compatible client at your endpoint through environment variables so the harness is not welded to one provider:
# redraft.py — example, credentials via env vars only import os from openai import OpenAI client = OpenAI( base_url=os.environ["MODEL_BASE_URL"], api_key=os.environ["MODEL_API_KEY"], ) def redraft(skeleton, fact_sheet, run_id): prompt = ( "Rewrite the outline as documentation. Use every fact exactly as given. " "Do not introduce numbers, identifiers, or defaults that are absent.\n\n" f"FACTS:\n{chr(10).join(sorted(fact_sheet))}\n\nOUTLINE:\n{skeleton}" ) resp = client.chat.completions.create( model=os.environ["MODEL_NAME"], messages=[{"role": "user", "content": prompt}], ) return resp.choices[0].message.content Record the run id, the temperature, and the exact prompt hash next to every output. Without those three fields you cannot tell a model behaviour change from a prompt edit you forgot about.
Step 4: Diff the claims into three buckets
For each output, extract claims with prose_facts and compare against the baseline. Three buckets matter: kept, dropped, and invented. Invented claims are worse than dropped ones because a reviewer reading for flow will not notice a plausible wrong default value.
def classify(baseline, drafted): return { "kept": sorted(baseline & drafted), "dropped": sorted(baseline - drafted), "invented": sorted(drafted - baseline), } def fidelity(baseline, drafted): if not baseline: return None # no measurable facts; exclude from ranking return len(baseline & drafted) / len(baseline) Exclude sections with no extractable facts instead of scoring them as perfect. A prose-only section that scores 1.0 because it had nothing to lose will distort your ranking.
Step 5: Rank sections and assign ownership
The table below is a proposed decision rule, not a measured result. Calibrate the bands against your own runs before you write them into a contribution guide.
| Fidelity across 3 runs | Invented claims | Suggested owner |
|---|---|---|
| 1.0 and stable | 0 | Model drafts, human spot-checks the diff |
| 0.9–0.99 | 0 | Model drafts, human edits against the dropped list |
| 0.7–0.89 | 0–1 | Human writes, model suggests phrasing only |
| below 0.7 or any run | 2 or more | Human owns, model excluded from the section |
Publish the resulting map next to the docs, with the date of the run. A stale fidelity map is more dangerous than none, because it grants a leash the code may no longer deserve.
Where free model access and a free server fit
This harness is cheap per section and expensive in aggregate, which is exactly the shape of workload that benefits from free model access and a free server option. One hundred and twenty redrafts across a documentation set is a trial, not a product feature, and paying per token to discover that your billing guide leaks error codes is a bad trade.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Two honest constraints apply. First, treat free-tier rate limits and availability as operational facts you must observe, not as guarantees to build a nightly CI job on. Second, do not send private source or internal documentation to any endpoint your team has not cleared for that data, free or paid.
A test plan for the harness itself
Before you trust any score, break a document on purpose. Take a section that currently passes, delete one default value from the prose, and confirm the diff reports it as dropped. Then add a plausible wrong status code and confirm it appears as invented. If both seeded failures are caught, your extractor is doing its job; if not, fix the patterns before running the real set.
Run the seeded test after every extractor change. Pattern edits are the most common way this harness silently stops measuring anything.
Limitations and who should skip this
Skip it for small documentation sets under roughly twenty sections, where review cost is lower than harness maintenance. Skip it where the facts live only in human heads or customer conversations, because there is no source to freeze a baseline from. Skip it for regulated or legal text where any model participation needs a compliance review you probably do not want to trigger.
The harness also cannot judge usefulness, ordering, or whether the page answers a real question. It measures one thing, fact survival, and a high score on that metric is a floor rather than a finish line.
If you already have a docs pipeline, the smallest next step is the seeded-failure test alone, since it costs an afternoon and tells you whether measurement is even possible in your repository.
