I used four specialised AI agents to turn 41 messy handover documents into four usable artifacts: two spreadsheets, a summary, and a 17-process training checklist.

The agents finished. The files existed. The row counts were right. The document opened.

Then I looked at the checklist.

Every numbered step continued from the previous section. Process 2 started where process 1 ended, and the document eventually ran from 1 to 169. The content was mostly fine. The structure underneath it was not.

A .docx is a ZIP of OOXML files, so I inspected word/document.xml instead of trying another prompt:

import zipfile with zipfile.ZipFile("Process Checklists.docx") as docx: xml = docx.read("word/document.xml").decode() print(xml.count("<w:numPr>")) # 0

The generated paragraphs used Word's shared ListNumber style, but none had an explicit numbering reference. The file was valid enough to open, while still being semantically wrong. Word treated the steps as one long list.

I rebuilt the document generator so every list that needed to restart received its own w:num instance, and every numbered paragraph received an explicit w:numPr reference.

The verification was deliberately structural:

paragraphs with explicit numPr: 318 distinct list instances used: 58 numbering instances declared: 67 orphan references: 0

I also replaced the flat heading hierarchy and emoji warnings with real heading styles, consistent callout blocks, page headers, footers, and restrained typography. That part was visual. The numbering bug was not; it needed an OOXML check.

The useful lesson was not “AI is bad at Word documents.” The agents did the expensive content work: cataloguing 41 files, building 17 procedures, and identifying nine gaps where the source notes were incomplete.

The failure was in the acceptance test. I had checked that the artifact existed and contained text. I had not checked that Word interpreted the document structure correctly.

For AI-generated office files, my minimum gate is now:

  1. verify the expected content and counts
  2. inspect the package structure for references and relationships
  3. render or open the artifact to check layout
  4. test the failure-prone feature directly

A generated file is not done because it opens. It is done when the application consuming it behaves correctly.