Split Your Docs by Claim Type, Not by Section Size: A Manifest That Decides What a Model May Draft

A page says the CLI flag --retries defaults to three. The code says five. The page has been wrong for four months, two reviewers read it, and nothing in CI noticed — because the sentence is prose, and prose has no exit code.

That is the actual problem with model-drafted documentation. It is not that a model wrote a paragraph. It is that the paragraph contained a claim nobody classified as checkable, so nobody checked it.

This article is about a small, boring mechanism that fixes that: a per-section manifest that assigns every heading one of three claim classes, plus a CI gate that enforces a different rule for each class. The model drafts only what the gate can defend.

Three claim classes, three different contracts

Most teams classify docs by topic (install, API, troubleshooting). Classification by topic tells you nothing about who may author it. Classify by what kind of claim the section makes instead.

The middle row is where most pages live, and it is also where the interesting failures hide. The bottom row is the one people forget — deprecation dates, security posture, compatibility guarantees. A model has no standing to promise anything on your behalf, and no gate can verify a promise. So the gate does the next best thing: it refuses the edit.

The artifact: a manifest next to the docs

One file per docs tree. Keep it flat and diffable.

# docs/claims.yaml version: 1 files: docs/install.md: sections: "Prerequisites": { class: descriptive, anchor: go.mod } "Install the CLI": { class: runnable, shell: "bash -euo pipefail" } "Verify the install": { class: runnable, shell: "bash -euo pipefail", timeout: 120 } "Versioning policy": { class: committal } docs/auth.md: sections: "Token scopes": { class: descriptive, anchor: internal/auth/scopes.go } "Data retention": { class: committal } "Rotate a key": { class: runnable, shell: "bash -euo pipefail" } 

Two rules make this cheap to adopt:

1. A heading that is missing from the manifest is a failure, not a default. Silence is how unclassified claims sneak in.
2. Runnables declare their shells per file, not per command, so a reviewer can see at a glance which pages touch the network.

The gate: ~70 lines, no dependencies beyond PyYAML

This is real, runnable code. It does three things: heading coverage, command execution with output matching, and trailer rejection for committal sections.

# tools/docs_gate.py import re, subprocess, sys, yaml, pathlib HEADING = re.compile(r"^(#{2,3})\s+(.+?)\s*$", re.M) FENCE = re.compile(r"``` (\w+)?\n(.*?) ```", re.S) TRAILER = re.compile(r"^AI-Assisted:\s*(\S+)", re.M) def sections(md: str): marks = [(m.start(), m.group(2)) for m in HEADING.finditer(md)] out = {} for i, (pos, name) in enumerate(marks): end = marks[i + 1][0] if i + 1 < len(marks) else len(md) out[name] = md[pos:end] return out def run_block(body: str, shell: str, timeout: int) -> tuple[bool, str]: for lang, block in FENCE.findall(body): try: p = subprocess.run([shell, "-c", block], capture_output=True, text=True, timeout=timeout) except subprocess.TimeoutExpired: return False, "timeout" if p.returncode != 0: return False, f"exit {p.returncode}: {p.stderr.strip()[:200]}" return True, "ok" def touched_by_model(section_body: str, base: str) -> bool: # Look for commits since `base` whose trailer marks model assistance  try: log = subprocess.run( ["git", "log", f"{base}..HEAD", "--format=%B", "--", "docs/"], capture_output=True, text=True, check=True).stdout except subprocess.CalledProcessError: return True # fail closed  return bool(TRAILER.search(log)) def main(base: str) -> int: manifest = yaml.safe_load(pathlib.Path("docs/claims.yaml").read_text()) errors = [] for rel, spec in manifest["files"].items(): md = pathlib.Path(rel).read_text() found = sections(md) declared = spec["sections"] for name in found.keys() - declared.keys(): errors.append(f"{rel}: unclassified section '{name}'") for name, meta in declared.items(): if name not in found: errors.append(f"{rel}: stale manifest entry '{name}'") continue cls, body = meta["class"], found[name] if cls == "runnable": ok, msg = run_block(body, meta.get("shell", "bash"), meta.get("timeout", 60)) if not ok: errors.append(f"{rel}#{name}: {msg}") elif cls == "descriptive": if not pathlib.Path(meta["anchor"]).exists(): errors.append(f"{rel}#{name}: anchor missing {meta['anchor']}") elif cls == "committal": if touched_by_model(body, base): errors.append(f"{rel}#{name}: model-assisted edit in committal section") else: errors.append(f"{rel}#{name}: unknown class {cls}") for e in errors: print(f"::error file=docs::{e}") return 1 if errors else 0 if __name__ == "__main__": sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "origin/main")) 

Wire it into CI with the merge base as the argument, so the committal check only inspects commits introduced by the branch:

BASE=$(git merge-base origin/main HEAD) python tools/docs_gate.py "$BASE" 

Run it locally before pushing and you get the same answer, which matters more than the CI wiring — a gate that only fires in CI trains people to bypass it.

Where the free model tier actually fits

The gate above deliberately does not care who wrote the prose. That is the point. The model is a drafting convenience for two of the three classes, and nothing more.

In my setup the drafting step looks like this: for each runnable or descriptive section, the harness sends the section body, the anchor file, and the last few diffs of that anchor, and asks for a revised body only. Committal sections are never in the prompt. I run that loop against MonkeyCode's free model access, and the operator also advertises a free server option — the operator describes a free tier of roughly ten million tokens plus a free server, which is enough to iterate on a docs tree without a procurement conversation. Quotas and terms change, so check the project's current site before you plan a workflow around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The project is described by its operator as open source, which is the property that matters here: I want the drafting harness to be inspectable, because I am asserting to my team that committal sections never reach a model. A closed pipeline cannot make that assertion cheap to verify.

If you want to try the pattern end to end, the sequence is: write claims.yaml for one page, get the gate green against the unmodified page, then turn the model loose on the runnable sections only. That ordering is not stylistic — it means the gate is proven before the model can produce anything the gate has to reject.

What this does not catch

Be honest about the boundaries before you ship it, or the gate becomes a talisman.

• Semantically wrong but executable output. A command that exits 0 while doing the wrong thing passes. Output matching helps; it is not proof.
• Stale anchors. The check is existence, not relevance. A symbol can move and still resolve to a file that no longer explains the paragraph.
• Trailer absence. The committal rule trusts your commit convention. Someone who omits AI-Assisted: defeats it. Pair it with a CODEOWNERS requirement on committal paths so a human signature is required regardless.
• Prose quality. Nothing here measures whether a paragraph is readable or useful. That stays a review problem.
• Network-dependent sections. Runnables that hit a live endpoint flake. Mark those with a longer timeout and expect noise, or move them to a nightly job.

Who should not use this

If your docs are fewer than roughly ten pages and change twice a year, the manifest is overhead you will resent — a single reviewer reading the whole page is cheaper and more reliable. If your docs are generated entirely from source annotations, the classification is already implicit in the code and this gate is redundant. And if you cannot run untrusted commands in CI safely, stop at the descriptive and committal rules; the runnable class assumes a sandbox you control.

The broader argument is the one worth keeping: the question is never "may an AI write our docs?" It is "which sentences in our docs can a machine defend, and who signs the rest?" A manifest that answers that per heading is a small file, and it survives the next model release without edits — which is the only kind of doc tooling that ages well.