More developers and buyers are discovering tools through ChatGPT Search, Perplexity, and Claude instead of clicking through ten blue links.
If you search for advice on "Generative Engine Optimization (GEO)," most articles immediately pitch vague content tips or $99/month prompt-tracking dashboards. Before worrying about prompt rankings, though, there is a much simpler question to answer:
Can AI crawlers even read your website's HTML?
After auditing dozens of developer tools, SaaS landing pages, and documentation sites, I kept finding the same four technical blockers. Here is how each one breaks AI visibility — and how to test your site in 10 seconds.
1. Blocking Live AI Search Bots in robots.txt
A lot of teams copied a blanket "block AI bots" snippet into robots.txt back in 2023 to opt out of model training. The problem is that AI platforms now separate training crawlers from live search & user-triggered fetchers.
If you want to opt out of training while still appearing as a live cited source in ChatGPT Search, Perplexity, or Claude, your robots.txt needs to distinguish them:
# Opt out of model training crawlers (optional) User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / # Allow live search and citation fetchers User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / Check your robots.txt right now — you might be blocking OAI-SearchBot or PerplexityBot without realizing it.
2. The Sneaky /llms.txt SPA 404 Trap
The llms.txt specification gives LLMs a clean, Markdown index of your site at /llms.txt (and optionally a full concatenated digest at /llms-full.txt).
Here is a bug that catches almost every Single-Page App (React, Vue, Vite, Cloudflare Pages, Netlify). Run this in your terminal against a typical SPA that doesn't have an llms.txt file yet:
curl -i https://example-spa.com/llms.txt Instead of 404 Not Found, the catch-all SPA router returns:
HTTP/2 200 content-type: text/html; charset=utf-8 <!doctype html> <html><body><div id="root"></div></body></html> Naive uptime or SEO checkers only check status === 200 and report "Valid llms.txt detected." An AI crawler, meanwhile, receives an HTML page instead of Markdown.
A real /llms.txt should return text/plain or text/markdown and follow a clean heading-and-link structure:
# Acme API > Acme API provides deterministic PDF invoice generation for Node.js and Python backends. ## Core Documentation - [Quickstart](https://acme.dev/docs/quickstart): Authenticate and generate your first PDF in 3 minutes. - [API Reference](https://acme.dev/docs/reference): Endpoints, request schemas, and webhook events. - [Pricing](https://acme.dev/pricing): Usage tiers and rate limits. 3. Empty Raw HTML (The JS-Only SPA Trap)
Googlebot runs a massive headless Chromium rendering pipeline. Most AI search fetchers do not — they optimize for fast, cheap HTTP fetches and parse the raw HTML response directly.
Test what a non-JS fetcher actually sees on your homepage:
curl -s https://yoursite.com | grep -o "<p>.*</p>" | head -n 5 If your raw HTML is mostly <script> bundles and an empty <div id="root"></div>, AI crawlers see almost zero product copy, pricing, or documentation text. Server-side rendering (SSR), static site generation (SSG), or pre-rendering critical marketing and docs routes fixes this completely.
4. Missing or Malformed JSON-LD & Citability Signals
Structured data isn't just for Google rich snippets. Clean JSON-LD (Organization, SoftwareApplication, Article, FAQPage) gives extraction pipelines unambiguous facts about what your tool does, its price, its documentation URL, and who maintains it.
Common mistakes I see in the wild:
- Trailing commas or unescaped quotes breaking
JSON.parse()inside<script type="application/ld+json">. - Missing canonical URLs causing duplicate docs versions to split citation signals.
- Marketing hero pages with vague taglines and no concrete factual headings.
Audit Any URL in 10 Seconds (Free & Open Source)
Because I got tired of running curl, checking seven robots.txt user-agents, inspecting raw HTML word counts, and validating JSON-LD by hand, I built GEO-Audit — a zero-dependency Node.js CLI and web auditor.
Option A — Test in your browser (no signup):
Paste any URL at https://geo-audit-c8g.pages.dev to get a 0–100 score across all 5 pillars plus an instant downloadable starter llms.txt draft.
Option B — Run locally with the MIT CLI (Node 20+, zero npm install):
git clone https://github.com/guyamos99/geo-audit cd geo-audit node cli/geo-audit.mjs https://yoursite.com It checks:
- All 7 major AI crawler rules in
robots.txt -
/llms.txt&/llms-full.txtpresence + spec compliance (including SPA HTML 404 trap detection) - SSR / raw HTML readability vs. JS-only shells
-
JSON-LDvalidity and entity coverage - Citability & metadata signals
If you maintain a large docs site or run client audits and want to automate this across up to 500 pages locally — with auto-generated llms.txt + llms-full.txt, white-label printable HTML reports, and a GitHub Action that opens PRs when your score or docs change — there's also an optional $29 one-time Pro toolkit (https://amosphere817.gumroad.com/l/geo-audit-pro).
Otherwise, the free CLI and web tool cover single-page checks completely. Run your site through it and drop your score (or any weird edge cases you hit) in the comments!
