A page can look complete in your browser and still deliver very little useful content in its initial HTML response.

That difference matters when a crawler reads the response without running the same JavaScript as your browser. The useful question is not whether a particular AI system always renders JavaScript—crawler behavior varies and can change. Ask instead: does the response contain the content you want a crawler to find?

Here’s a quick way to check.

1. Fetch the page as HTML

Use a real page URL and a crawler-style user agent:

curl -sS -A "GPTBot" https://your-domain.com/page/ -o page.html

This shows what the server returned for that request. It does not prove that every crawler receives the same response, but it gives you a useful first check.

2. Look for the text that matters

Search the response for a phrase that appears in the rendered page:

grep -i -n "a phrase from the page" page.html

If the phrase is missing, compare the response with the page you see in your browser. Check whether the content is fetched only after the page loads, added after scrolling, or hidden behind a client-side route.

You can also get a rough word count after stripping HTML tags:

sed 's/<[^>]*>/ /g' page.html | tr -s ' ' | wc -w

Treat this as a quick diagnostic, not a precise content measurement. Navigation, footer text, scripts, and other markup can affect the result.

3. Compare with JavaScript disabled

In your browser’s developer tools, disable JavaScript for the page and reload it. Then compare what remains with the normal rendered page.

If the article disappears and only the navigation or an empty app shell remains, the content may depend on client-side code for its first render. That is worth investigating, especially for pages you want crawlers to read directly.

Common causes

  • Client-side fetching: the first response contains a loading state, while the article arrives later.
  • Scroll-triggered loading: content is fetched only after a reader scrolls to it.
  • Client-side routing: a deep link returns an app shell without the article in the initial response.
  • JavaScript-injected structured data: JSON-LD is added after the response instead of being present in the HTML.

These patterns can work well for people using a browser. The issue is whether the first response also contains the content a crawler needs.

What this check can—and cannot—tell you

A page with useful text in its initial HTML is easier to inspect and does not depend on a rendering pass just to expose its main content. That is a good baseline.

It does not guarantee a citation, recommendation, or ranking. A crawler may use other signals, and an AI answer engine may choose another source. Even a simple automated word-count check has limits: it may count navigation and footer text rather than the article itself.

For an important page, inspect the actual response and confirm that its main text and structured data are present there. Repeat the check against the deployed URL after changes, because a CDN, consent layer, or bot rule can alter what the origin serves.

I maintain LLMention, a scanner that checks whether a homepage is accessible and readable across a published set of rules. It measures crawl and citation readiness; it does not ask AI systems whether they mention a brand. The complete walkthrough and its limitations are on the original article.

The practical rule is simple: if the content you care about is absent from the response body, do not assume a crawler will reconstruct it the way your browser does.