An agent rebuilds your PDF letterhead: what Opus on high effort, Opus on low, a cheap model and a local 9B make of it

Kumiko has a content editor for PDF and mail templates: invoices, letters, statements. Nobody wants to rebuild their letterhead by hand in a new editor, so we added an import. You upload the PDF you use today and an agent rebuilds it as a normal, editable template. It looks at the pages as images, puts header, footer, logo, margins, fonts and paper into a layout, writes the body with placeholders, renders a preview, compares it with the original and corrects up to three times. At the end it shows a card with the original next to the draft and two lists: what stayed fixed text, and what it couldn't reproduce.

The agent doesn't write HTML. It calls the same write handlers the editor uses (create a layout, patch a draft, upload a resource), and everything it writes stays a draft until a person publishes it.

The setup

Three made-up fixtures, modelled on real templates, with invented names, IBANs and amounts:

• a club invoice with an embedded logo, address window, item table with sub-lines, subtotals and a three-column footer,
• a two-page bank tax statement with a vector logo, corner and fold marks, a monospace body next to a sans-serif footer, and a running header on page 2,
• a garage invoice delivered as a scan with no text layer, with coloured waves at the top and bottom edge and cream paper.

Each run goes through the real stack: HTTP, the agent's turn stream, Postgres, and the puppeteer service that rasterises PDFs and renders previews. The benchmark may say "go on" twice and approves the logo upload the way a user would. It records tool calls, LLM calls, tokens, cost, wall time and SSIM, and writes the original and the draft side by side into one image. Two runs per template and model; Opus on high effort ran in two rounds, before and after the last editor fixes.

Our first grades were too kind

The first write-up of this measurement gave Opus grades between 1-2 and 2, on the German school scale where 1 is best and 6 is worst. Then we put the pictures next to each other properly. The coloured areas were missing, the waves were missing, table lines, the logo position, the header data on page 2. Nothing deserved better than a 3.

Most of it wasn't the model. The editor couldn't express what a normal business letter has. An image had no print width and no alignment, so a right-aligned logo ended up on the left. Tables had no cell padding, no width and no line above a total. There were no fold or corner marks, no header that starts on page 2, no paper colour.

So we did what the plan had asked for from the start and skipped: rebuild all three templates by hand in the editor, without any AI, and put each one next to its original. Every gap that showed up became an editor feature. Only when the hand-built versions matched did the models run again.

From now on we only write down checklists per template: what is there, what is missing. The grade comes from the person who owns the product, looking at the pictures.

The models

• Claude Opus 5.5 on our own Anthropic key, with effort high (the default) and with effort low
• gpt-6-luna through OpenRouter, as the cheap option
• Qwen 3.5 9B on a Mac through Ollama, as the self-hosted option

Opus on high effort

Opus rebuilds the club invoice with the logo on the right, the sender line, the info block, items with their sub-lines, subtotals, a bold total, the IBAN in bold in the payment text and the three-column footer. On the tax statement it places the bank address and the date block right-aligned, sets corner marks, rules the table and puts the page number into the footer. On the scan it cuts the waves out of the page image as two pictures, places them at the paper edge and reads the cream paper colour from the page margin.

The waves and the logos needed a new tool option. Our image tool could only hand over images that sit in the PDF as embedded files. A scan is one big image and a vector logo is drawing commands. So the tool got a crop option: a page and a box in pixels of the page image the model has already seen. The service cuts the box out and stores it as the logo.

What's left: three fold marks on the tax statement where the original has one, table text a shade lighter than the original, and a logo a few millimetres too high on the club invoice.

Left: the scanned original. Right: Opus on high effort. The preview data has three line items, the original five.

Opus on low effort

Low costs about half. On the letters you'd hardly see a difference: titles and table headers come out a bit larger and bolder, and the running header on page 2 sits higher than it should. On the scan the difference is large. With low effort Opus made five tool calls, never rendered a preview and never compared. The header wave is missing, the paper is white, and the title overlaps the address block.

The import is a one-off setup step, so we leave it on high. In Kumiko the import isn't a separate agent. It is a set of tools inside the normal assistant, so it runs with the effort the tenant has configured, and the factory default is high.

Same scan, Opus on low effort: no waves, white paper, title on top of the address block.

luna

luna is cheap: 3 to 10 cents per template. It gets content and placeholders right and struggles with everything around them. Line spacing in the header doesn't match, footers fall apart, and on one run it asked questions until it ran out of turns. For a rebuild that should look like the original, that is not usable.

Same scan, gpt-6-luna: the waves are there, but title, address and vehicle table land on top of each other.

Qwen 9B on a Mac

0 of 3, in two rounds. Two runs hit the provider timeout as soon as the page images arrived. The others ran out of steps with text answers and malformed tool calls.

A 9B model on a laptop is slow with images, and the task needs vision, a long prompt and about twenty correct tool calls in a row. At this size it doesn't work. Bigger self-hosted models with vision and tool use, for example on a GPU behind vLLM, we haven't measured.

Two runs that ended in an error

Two of 18 Opus runs ended with agent.tool_not_allowed, both times after the draft was already done. Our agent doesn't hand the model every tool. It gets a directory of actions plus two fixed tools, describe_action and run_action, and run_action is unwrapped into the named action. If the name isn't in the directory or the input isn't an object, the call stays run_action, and the agent ends the turn rather than letting a model keep writing after it asked for something that doesn't exist.

We don't know yet which of the two it was. run_action now also accepts the input as a JSON string, which models sometimes send for large objects, and the benchmark logs the action name and the type of the input, so the next case will show the cause.

Where the money goes

A high-effort import costs $1 to $1.80. Most of it is the cache: every LLM call sends the whole conversation again, page images included, and a run makes 15 to 25 calls. Against a tenant's monthly AI budget on our platform key, one import counts 120,000 to 170,000 tokens. For something you do once per template, that's fine.

SSIM doesn't help

We hoped image similarity would do the grading. Finished rebuilds scored between 0.66 and 0.83. When a run failed and left the starting template untouched, that blank template scored 0.77 to 0.84, higher than most rebuilds. White paper dominates the score. The verdicts come from looking at the pictures.

What this doesn't prove

Three templates, two runs each, judged by eye, and the editor changed between rounds, so early and late rounds aren't a clean before and after. One limit sits in the product: placeholders in a layout header. One layout serves many templates, so "tax year 2025 · depot 0000" in the running header stays fixed text.

An open question

The result card lists what stayed fixed text, so real customer data from the upload doesn't end up in the template unnoticed. How do you get a person to actually read that list before they publish?