Keep the Prose, Re-Check the Nouns: A Token-Level Staleness Gate for Model-Drafted Docs

Most documentation pipelines I see spend all their guardrails on drafting and none on maintenance, which is exactly where model-written pages quietly decay. The conclusion up front: let a model draft freely, but force every sentence to carry its checkable nouns in backticks, then re-verify those nouns against the repository instead of re-reading the prose. Sentences whose tokens still resolve stay untouched for months, while only sentences with broken tokens return for a rewrite and a human signature. Below is that loop, a reference checker, the rewrite contract, and the cases where the whole idea collapses.

Why token-level beats page-level regeneration

Page-level regeneration is easy to schedule and expensive to review, because one renamed flag can rewrite forty paragraphs nobody needed to re-read. Reviewers then approve the diff by skimming, and the skim is precisely where a wrong default value survives into production docs. Token-level checking inverts the cost: a rename invalidates three sentences, the diff touches three sentences, and the reviewer reads three sentences carefully.

The unit of trust changes too. A page-level pipeline asks "is this page still true", a question no cheap tool can answer. A token-level pipeline asks "does this identifier, flag, path, or environment variable still exist in the source tree", which grep answers in milliseconds and which nobody has to accept on faith.

Step 1 — Sort facts into machine-checkable and human-owned

Do this before writing code, because the split determines what the model is ever allowed to touch. I use four classes, and only the first two are automated.

The last two rows are the ones a model must never invent, and they are also the rows reviewers care about most. A page with no backticked nouns gains nothing from this workflow, so start by adding them rather than by generating more text.

Step 2 — Put the owner in the front matter

Each page declares one accountable human and a staleness budget, so the checker has a destination for its failures.

--- owner: @avery-lin reviewed: 2026-11-06 stale_budget: 0 --- 

Ownership can be split finer if your team insists, but one owner per page keeps the escalation path short. A page without an owner line fails the check by definition, which is the correct default for documentation nobody has claimed.

Step 3 — The checker as a reference implementation

The script reads a page, extracts backticked tokens per sentence, and asks grep whether each token still exists in the configured paths. Nothing here is model-specific, and the standard library is sufficient. Treat the regexes as a starting point rather than a finished tool, since I have not benchmarked this against a large public repository.

#!/usr/bin/env python3 """docstale.py - flag doc sentences whose backticked tokens no longer resolve.""" from __future__ import annotations import json import re import subprocess import sys from pathlib import Path TOKEN_RE = re.compile(r"`([^`\n]{1,80})`") SENT_RE = re.compile(r"(?<=[.!?])\s+") MATTER_RE = re.compile(r"\A---\n(.*?)\n---\n", re.S) def split_front_matter(text: str) -> tuple[dict, str]: m = MATTER_RE.match(text) if not m: return {}, text meta = {} for line in m.group(1).splitlines(): if ":" in line: key, _, value = line.partition(":") meta[key.strip()] = value.strip() return meta, text[m.end():] def resolves(token: str, paths: list[str]) -> bool: cmd = ["grep", "-rFl", "--", token, *paths] return subprocess.run(cmd, capture_output=True).returncode == 0 def main(page: Path, manifest: Path) -> int: cfg = json.loads(manifest.read_text()) ignore = set(cfg.get("ignore", [])) meta, body = split_front_matter(page.read_text()) owner = meta.get("owner", "UNASSIGNED") budget = int(meta.get("stale_budget", 0)) findings, checked = [], 0 for raw in SENT_RE.split(body): tokens = [t for t in TOKEN_RE.findall(raw) if t not in ignore] if not tokens: continue checked += 1 missing = [t for t in tokens if not resolves(t, cfg["paths"])] if missing: findings.append({"owner": owner, "missing": missing, "sentence": raw.strip()}) report = { "page": str(page), "owner": owner, "sentences_checked": checked, "stale": len(findings), "budget": budget, "findings": findings, } print(json.dumps(report, indent=2)) return 1 if len(findings) > budget else 0 if __name__ == "__main__": sys.exit(main(Path(sys.argv[1]), Path(sys.argv[2]))) 

The manifest is deliberately boring, because it is the only file a reviewer must audit when the checker disagrees with reality.

{ "paths": ["src", "migrations", "docs/openapi.yaml", ".env.example"], "ignore": ["true", "false", "null", "GET", "POST"] } 

Run it locally before trusting it in CI, and read the JSON rather than only the exit code.

python3 docstale.py docs/guides/sessions.md docstale.json; echo "exit=$?" 

Step 4 — Give the model a narrow rewrite contract

Only flagged sentences are eligible, and the prompt forbids new tokens. That single constraint prevents a rewrite from quietly re-describing an API the model has never seen, which is the failure mode that scares most reviewers.

Rewrite ONLY the sentences in findings[]. Rules: 1. Keep every token that still resolves byte-for-byte. 2. Replace only tokens listed in findings[].missing, using the current source. 3. Introduce no new backticked tokens and no new numbers. 4. Leave the front matter untouched. Return one diff hunk per sentence; no commentary. 

This is where free model access is genuinely useful rather than decorative, because the job is small, repetitive, and strictly bounded to one sentence at a time. Disclosure: This article was prepared as part of MonkeyCode's product outreach. In this loop that rewrite step is bursty and idempotent, so it fits a free server run better than an interactive session; check the current free-model and free-server terms yourself before scheduling anything against them.

Step 5 — Wire the gate and keep the human step

Run the checker in CI on any pull request touching a documented path, and fail the build when a page exceeds its budget. When the rewrite job produces a diff, the owning human approves only that diff, and their approval is what updates the reviewed date. A passing checker is evidence about nouns, never evidence that the surrounding paragraph still makes sense.

Limitations and who should skip this

Tokens outside backticks are invisible to the checker, and paraphrases such as "the session endpoint" evade it entirely, so prose discipline is a prerequisite rather than an afterthought. grep also proves existence, not meaning: a constant can keep its name while its units change from seconds to milliseconds, and the check stays green. Diagrams, screenshots, and tutorials whose steps live in images fall outside the loop and still require scheduled human review.

Skip this workflow if your docs are mostly narrative, if CI cannot search the source tree, or if pages still lack named owners; a token checker with no owner produces a green dashboard over unowned pages. If you want to try the loop cheaply, run the rewrite step on the free server option and keep the verification step on your own CI runner, where the source of truth already lives.