I spent the last few weeks wiring document feeds into agent context windows, and the thing that keeps surprising me isn't the clean PDFs — it's everything else.
My test fixtures were all hand-picked, well-formed files. Real inputs are scanned contracts with rotated pages, spreadsheets where three sheets share a header row, HTML pages where the main article is buried under nav and cookie banners. Every one of those broke something at first: tables came out as comma soup, boilerplate leaked into the "main content," merged cells silently dropped rows.
The lesson I keep re-learning: optimize for the mess, not the sample. When I built the pipeline that auto-detects PDF/DOCX/PPTX/XLSX/EPUB/HTML and returns clean Markdown in one call (https://x402.freeq.one/tools/document_to_markdown.html), the detection part was easy. The hard part was accepting that "good enough" extraction has to be measured against real documents, not the ones I curated.
Practical takeaways if you're shipping something similar:
- Log and inspect a sample of actual outputs, not just unit tests. Fixtures lie to you.
- Give the caller control (selectors, format options, length caps) — you can't anticipate every layout, but the agent downstream can.
- Treat token budget as a feature. Stripping sidebars and nav isn't cosmetic; it's what makes a page fit in a context window at all.
Garbage in, garbage out still holds — the fix isn't a smarter model downstream, it's cleaning at the boundary.

