If you maintain developer documentation, an API platform, or a SaaS web app in Next.js (App Router), adding machine-readable endpoints for AI coding assistants (Cursor, Windsurf, GitHub Copilot, Claude Code) and AI search engines (OAI-SearchBot, PerplexityBot) takes less than 10 minutes.

In this tutorial, we will look at the exact structure required by the llmstxt.org specification, the 3 most common RFC 9309 robots.txt mistakes that break AI crawlers, and how to serve /llms.txt and /.well-known/tdmrep.json using Next.js static route handlers.

1. The Strict llmstxt.org Markdown Grammar

Unlike sitemap.xml, /llms.txt is designed to be read by both deterministic parsers and LLM tokenizers. The specification enforces a strict ordering:

# Project or Platform Name  > A concise, factual 1-to-2 sentence summary of what the platform does, its core architecture, and target audience (zero marketing superlatives).  Optional concise operational notes (e.g., authentication model, rate limits, or API base URL). ## Core Product & Tools  - [JSON Formatter & Validator](https://example.com/developer/json-formatter): Strict RFC 8259 JSON validator with 64-bit BigInt truncation detection and JSONPath evaluation. - [HAR File Analyzer & Sanitizer](https://example.com/developer/har-analyzer): Client-side 7-phase HTTP waterfall inspector and 14-surface secret scrubber. ## Optional  - [Privacy Policy](https://example.com/privacy): Details on client-side data handling and GDPR/CCPA compliance.

Three Linter Rules to Watch Out For:

  1. Exactly One # H1 at Line 1: Parsers expect the document to begin with a single H1 heading followed immediately by the > summary blockquote.
  2. No Vague Link Anchors: Never write - [Click here](...) or - [Read more](...). The anchor text and colon-separated description are what the LLM uses to decide whether to fetch the underlying URL.
  3. Use the ## Optional Section for Secondary Pages: According to the spec, links under ## Optional tell context-constrained models which URLs can be skipped when token budgets are tight.

2. Serving /llms.txt in Next.js App Router (app/llms.txt/route.ts)

In Next.js App Router, you can serve /llms.txt at the edge with zero runtime overhead using force-static and Cache-Control headers:

// app/llms.txt/route.ts export const dynamic = 'force-static'; const LLMS_TXT_CONTENT = `# YourPlatformName > Factual summary of your platform and technical architecture. ## Docs & Core Tools - [API Reference](https://example.com/docs/api): Complete REST & GraphQL endpoint specifications. `; export async function GET() { return new Response(LLMS_TXT_CONTENT, { status: 200, headers: { 'Content-Type': 'text/plain; charset=utf-8', 'Cache-Control': 'public, max-age=3600, s-maxage=86400, stale-while-revalidate=604800', 'X-Robots-Tag': 'all', }, }); }

3. Auditing Your robots.txt for RFC 9309 Prefix Leaks

Under IETF RFC 9309, Disallow: /admin matches /admin, /admin/, AND /admin-guide.html (because Disallow is a prefix match!). Conversely, if you write Disallow: /private/, a request to /private (without the trailing slash) is still allowed.

And if you use Cloudflare in front of Next.js, verify that Cloudflare's Managed robots.txt ("Block AI training bots") isn't prepending its own rules above your app/robots.ts output—which frequently causes inconsistent AI bot policies across Amazonbot, Claude-Web, and Meta-ExternalAgent.

If you want to auto-convert your existing sitemap.xml or HTML docs into llms.txt, llms-full.txt, W3C tdmrep.json, and Next.js route code—while profiling token usage across GPT-4o (128k), Claude 3.7 Sonnet (200k), and DeepSeek R1 (64k)—you can use the client-side KivaroTools llms.txt & AI Crawler Generator.