AI agents read web pages the expensive way. They download the HTML, throw away the navigation, scripts and styles, and try to recover the text. Some of them now send Accept: text/markdown and take Markdown if the server offers it.

I wanted our landing page to answer that request without keeping a second Markdown copy of the site in sync by hand. The site is static files behind a Cloudflare Worker, so the Worker converts the HTML on the way out. This post walks through that module: about 160 lines, no dependencies, built on HTMLRewriter.

Cloudflare Workers edge network

Technical diagram of content flow

Step 1: decide when to convert

The rule is narrow on purpose. Convert only a successful GET for an HTML asset, and only when the client actually asked for Markdown:

export async function serveMarkdownForAgents(request, assets) { const response = await assets.fetch(request); if (request.method !== 'GET' || !wantsMarkdown(request) || !response.ok || !response.headers.get('content-type')?.toLowerCase().startsWith('text/html')) { return response; } // ...convert }

Images, CSS, the sitemap and everything else pass through untouched, even when the request prefers Markdown. Browsers never send text/markdown, so people keep getting HTML.

wantsMarkdown reads the Accept header properly instead of searching it for a substring. It splits the media ranges, looks for text/markdown, and respects the quality value:

function wantsMarkdown(request) { return request.headers.get('accept')?.split(',').some((part) => { const [mediaType, ...parameters] = part.trim().toLowerCase().split(';'); if (mediaType !== 'text/markdown') return false; const quality = parameters.find((parameter) => parameter.trim().startsWith('q=')); if (!quality) return true; const value = Number(quality.trim().slice(2)); return value > 0 && value <= 1; }) ?? false; }

The q=0 case matters. Accept: text/html, text/markdown;q=0 means "anything but Markdown", and a naive includes('text/markdown') would send exactly the wrong thing. There is a test for that case, and one confirming that a non-HTML asset is unchanged when Markdown is requested.

Step 2: convert with HTMLRewriter

HTMLRewriter is a streaming parser. You register handlers for selectors and get callbacks for elements and text chunks. There is no DOM tree, so the converter keeps a little state of its own:

  • an output array of string chunks;
  • a stack of list counters, so <ol> items get 1., 2. and <ul> items get -;
  • a skippedDepth counter for subtrees that shouldn't appear at all.

Everything that is page chrome is skipped: head, nav, footer, script, style, noscript, svg, button, template, plus anything marked hidden, aria-hidden="true", role="navigation" or with a skip-link class. When such an element opens, the depth goes up, and onEndTag brings it back down. Text callbacks only write when the depth is zero.

if (OMIT.has(tag) || hidden) { element.remove(); if (!VOID.has(tag)) { skippedDepth += 1; element.onEndTag(() => { skippedDepth -= 1; }); } return; }

The VOID check is easy to miss. Void elements such as <img> or <br> have no end tag, so registering onEndTag on them would leave the counter stuck and silently drop the rest of the page.

The mapping itself is ordinary. Headings become # prefixes. Paragraphs and sections become blank lines. strong and em become ** and *, and pre becomes a fenced block. Two cases need care:

  • Links are rewritten to absolute URLs, and only http, https, mailto and tel survive. A relative /privacy becomes https://.../privacy, which an agent can follow, and a javascript: link is dropped.
  • Images appear only when they have both alt text and a resolvable src. A decorative image without alt text carries nothing an agent can use.

Text nodes are decoded (&amp;, &nbsp;, numeric entities) and whitespace is collapsed. After the stream ends, one cleanup pass trims spaces around newlines and caps blank lines at one.

Step 3: keep the metadata

The parts of <head> that matter to an agent are kept, just not as HTML. Separate handlers collect the <title>, the meta description, the canonical URL and the text of any application/ld+json blocks. The output starts with front matter:

--- title: "..." description: "..." url: "https://example.com/" ---

The JSON-LD is appended at the end under a ## Structured data heading. The structured data a search engine sees, such as the FAQ and the product details, also reaches an agent that never parses HTML.

Step 4: fix the headers

This is the part that breaks caches if you skip it. The body changed, so several headers copied from the HTML asset are now lies:

headers.set('content-type', 'text/markdown; charset=utf-8'); // add Vary: Accept unless it is already there (or Vary: *) for (const name of ['content-length', 'content-encoding', 'content-md5', 'etag', 'content-range', 'accept-ranges', 'digest']) headers.delete(name);

Vary: Accept is what stops a shared cache from serving the Markdown version to the next browser that asks for the same URL. Dropping the old ETag avoids the opposite mistake: a conditional request answered with a 304 that points at the HTML body.

Last, the whole conversion is wrapped in try/catch. If anything throws, the original HTML response goes out. An agent getting HTML is a degraded result; a 500 is a broken one.

Trying it 

curl -s -H "Accept: text/markdown" https://talkaside.com/ | head -20

The same URL in a browser still returns the normal page.

Limits

The converter is tuned to this site's semantic HTML, and the source says so in a comment. Tables come out as pipe-separated rows without the |---| header separator, so they aren't strictly valid Markdown tables. Anything rendered by JavaScript after load isn't there, because the Worker only sees the static HTML. A site with rich widgets or complex tables would be better served by a real HTML-to-Markdown library, at the cost of a bundled dependency.

For a marketing page made of headings, paragraphs, lists and an FAQ, 160 lines of HTMLRewriter were enough, and there is only one source of truth.

I'm building Talkaside, a Windows app for job interviews and calls in another language.

Drafted with AI assistance from this project's code and logs.