More and more users get their answers from ChatGPT, Perplexity, Gemini or Google's AI Overviews instead of clicking a blue link. Pew Research found people clicked a traditional search result in about 8% of visits with an AI summary, versus 15% without one.

Marketers call the response generative engine optimization (GEO). A lot of GEO is content strategy, but a surprising amount is plain engineering: can AI crawlers reach your pages, read them, and understand them?

Here's the developer side of it.

1. Check your robots.txt for AI crawlers

Many sites block AI bots without knowing it, through a blanket rule, a CDN "bot protection" setting, or a copy-pasted robots.txt from 2023.

These are the common user agents to know:

User agent Operator Purpose
GPTBot OpenAI Model training
OAI-SearchBot OpenAI ChatGPT search results
ChatGPT-User OpenAI Fetches pages when a user asks
ClaudeBot Anthropic Crawling
PerplexityBot Perplexity Search index
Google-Extended Google Gemini training opt-out token

A setup that allows AI search but opts out of training might look like this:

User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Disallow: / 

Two gotchas:

  • Google-Extended does not control whether you appear in AI Overviews. Those come from regular Googlebot crawling.
  • Your CDN or WAF can block bots even if robots.txt allows them. Check Cloudflare's (or your provider's) bot settings too.

2. Don't hide your content behind JavaScript

Googlebot renders JavaScript, but many AI crawlers fetch the raw HTML and move on. If your article body only appears after client-side hydration, some AI engines see an empty page.

Quick test:

curl -s -A "PerplexityBot" https://yoursite.com/your-post | grep -c "<p>" 

If that returns 0 (or very few), your content probably isn't in the initial HTML. Fixes:

  • Use SSR or static generation (Next.js, Astro, Nuxt, etc.) for content pages
  • Make sure the main text, headings and key facts are in the server response
  • Don't lazy-load the actual article body

3. Check your server logs: are AI bots visiting at all?

Before optimizing anything, find out who is actually crawling you:

grep -Ei "gptbot|oai-searchbot|chatgpt-user|claudebot|perplexitybot" access.log \ | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head -20 

Look for:

  • No hits at all: something is blocking them
  • Lots of 403s or 429s: your WAF or rate limiter is rejecting them
  • Hits only on your homepage: your internal linking or sitemap needs work

4. Add structured data

JSON-LD helps machines understand what a page is: an article, a product, a how-to, an organization. At minimum, add Article and Organization schema:

<script type="application/ld+json"> { "@context": "https://schema.org", "@type": "Article", "headline": "Your article title", "datePublished": "2026-11-17", "dateModified": "2026-11-17", "author": { "@type": "Person", "name": "Your Name" }, "publisher": { "@type": "Organization", "name": "Your Company" } } </script> 

Keep dateModified accurate. Freshness matters for AI answers on fast-moving topics.

5. Structure content so it can be quoted

AI engines pull out passages, not whole pages. Make yours easy to pull out:

  • Use question-style headings (## What is X?) followed by a direct 1–2 sentence answer
  • Use semantic HTML: real <h2>/<h3>, <table> for tabular data, <ol> for steps
  • Include specific facts, numbers and cited sources

That last point has research behind it: a KDD 2024 paper on GEO found that adding citations, quotations and statistics could raise visibility in AI-generated answers by up to 40%.

6. Consider llms.txt (but don't count on it)

llms.txt is a proposed standard: a markdown file at your site root that gives LLMs a curated map of your most important content. It's cheap to add, but support from the major AI engines is unclear, so treat it as a nice-to-have, not a fix.

7. Measure it

You can't use Search Console to see how often ChatGPT cites you. That's where GEO tools come in. They run a set of prompts across AI engines and track whether your brand gets mentioned and which sources get cited.

If you're comparing options, I wrote up a detailed breakdown of pricing, engine coverage and team fit: 11 Best Generative Engine Optimization Tools

On a budget? Start by checking your server logs and manually asking a few AI engines your target questions every week. It's crude, but it works.

TL;DR checklist

  • [ ] robots.txt allows the AI search bots you want
  • [ ] CDN/WAF isn't silently blocking them
  • [ ] Content is in the initial HTML (SSR/SSG)
  • [ ] Server logs show AI crawlers getting 200s
  • [ ] Article + Organization JSON-LD in place
  • [ ] Question-based headings with direct answers
  • [ ] Facts, stats and sources in your content
  • [ ] Some way to track AI visibility over time

What's your setup? Are you allowing AI crawlers or blocking them? Let me know in the comments.

Disclosure: I work at Blazly, one of the tools covered in the linked comparison.