Why Your Next.js Page Returns 200 OK but Still Isn't Indexed

A page can load perfectly in your browser, return 200 OK, and still be absent from Google Search.

That situation is frustrating because the usual first check—opening the URL and confirming that it works—does not test the things a crawler needs to discover, render and understand the page.

For a Next.js App Router site, I would debug this in layers rather than immediately changing metadata or requesting indexing. The goal is to establish what the server actually returns, what the browser renders, what Google is allowed to fetch, and which URL the page claims as its canonical version.

This guide walks through that process with commands and code you can adapt to a real project.

First, understand what 200 OK tells you

An HTTP 200 response means that the server successfully handled the request. It does not mean that:

• the page contains its intended content in the initial HTML;
• search engines are allowed to index it;
• the page has a valid canonical URL;
• Google has rendered the JavaScript successfully;
• the page is included in the sitemap;
• Google has selected the URL for indexing.

Google describes JavaScript Search processing as three stages: crawling, rendering and indexing. A page can pass one stage and still have a problem at another. Google generally queues crawlable 200 pages for rendering, but rendering is a separate step and may not happen immediately. Google's JavaScript SEO guide explains the process in more detail.

The practical consequence: treat 200 OK as the beginning of the investigation, not the conclusion.

1. Check the actual HTTP response

Start with the exact URL that is missing from search. Do not test only the homepage or a nearby route.

On macOS, Linux, WSL or Git Bash, run:

URL="https://www.example.com/services/it-support" curl -sS -L \ -D response-headers.txt \ -o response-body.html \ -w '\nFinal status: %{http_code}\nFinal URL: %{url_effective}\n' \ "$URL" 

This follows redirects, saves the response body and records the response headers. The header file can contain more than one response block when redirects occur.

Inspect the relevant headers:

grep -Ei '^(HTTP/|location:|content-type:|x-robots-tag:)' response-headers.txt 

On Windows PowerShell, call curl.exe explicitly to avoid the older curl alias:

$Url = "https://www.example.com/services/it-support" curl.exe -sS -L -D response-headers.txt -o response-body.html `  -w "`nFinal status: %{http_code}`nFinal URL: %{url_effective}`n" $Url Select-String -Path .\response-headers.txt -Pattern '^(HTTP/|location:|content-type:|x-robots-tag:)' 

Look for these outcomes:

A useful test is to compare the response for a real page with the response for a deliberately invalid URL. If both return the same status and effectively the same page, your not-found handling may be wrong.

In the App Router, use Next.js not-found handling for missing resources instead of rendering a normal-looking error message with a successful response. See Next.js error handling and not-found routes.

2. Inspect the HTML the server sends before JavaScript runs

This is one of the highest-value checks for a JavaScript-heavy website.

Open response-body.html and search for the page's actual main heading and copy. You can also run:

grep -oi '<title[^>]*>[^<]*</title>' response-body.html grep -oi '<h1[^>]*>[^<]*</h1>' response-body.html grep -oi "<meta[^>]*name=[\"\']robots[\"\'][^>]*>" response-body.html grep -oi "<link[^>]*rel=[\"\']canonical[\"\'][^>]*>" response-body.html 

These quick searches are useful, but HTML can be minified, multiline or ordered differently. For a more reliable first-pass inspection, here is a small Python script using only the standard library.

Save this as inspect_html.py:

from html.parser import HTMLParser from pathlib import Path import sys class PageInspector(HTMLParser): def __init__(self): super().__init__(convert_charrefs=True) self.title_parts = [] self.h1_parts = [] self.links = [] self.robots = [] self.canonicals = [] self._in_title = False self._in_h1 = False def handle_starttag(self, tag, attrs): attrs = dict(attrs) if tag == "title": self._in_title = True elif tag == "h1": self._in_h1 = True elif tag == "meta": name = (attrs.get("name") or "").lower() http_equiv = (attrs.get("http-equiv") or "").lower() if name == "robots" or name == "googlebot": self.robots.append(attrs.get("content", "")) if http_equiv == "refresh": self.robots.append("META REFRESH: " + attrs.get("content", "")) elif tag == "link": rel = (attrs.get("rel") or "").lower().split() if "canonical" in rel: self.canonicals.append(attrs.get("href", "")) elif tag == "a" and attrs.get("href"): self.links.append(attrs["href"]) def handle_endtag(self, tag): if tag == "title": self._in_title = False elif tag == "h1": self._in_h1 = False def handle_data(self, data): if self._in_title: self.title_parts.append(data) if self._in_h1: self.h1_parts.append(data) def main(): if len(sys.argv) != 2: raise SystemExit("Usage: python inspect_html.py response-body.html") html = Path(sys.argv[1]).read_text(encoding="utf-8", errors="replace") parser = PageInspector() parser.feed(html) print("Title:", " ".join(" ".join(parser.title_parts).split()) or "NOT FOUND") print("H1:", " ".join(" ".join(parser.h1_parts).split()) or "NOT FOUND") print("Robots directives:", parser.robots or "None found in HTML") print("Canonical URLs:", parser.canonicals or "None found in HTML") print("Anchor links found:", len(parser.links)) for href in parser.links[:25]: print(" -", href) if __name__ == "__main__": main() 

Run it:

python inspect_html.py response-body.html 

This script is deliberately a diagnostic aid, not a complete HTML validator or a Googlebot emulator. It reports what is present in the downloaded response. It does not execute JavaScript, resolve relative URLs or decide whether a page should be indexed.

What if the HTML contains only an app shell?

Suppose the server response contains a root element and JavaScript bundles, but none of the page's main text. The page may rely on client-side fetching to render its content.

Google can render JavaScript, so an empty initial response does not automatically prove that the page cannot be indexed. It does mean you have added another dependency: the renderer must fetch the scripts, execute them, obtain the data and produce the content. It also creates problems for crawlers that do not execute JavaScript.

For important public pages, prefer server-rendered or statically generated content where it makes architectural sense. In Next.js, Server Components are the default in the App Router; use Client Components for the interactive parts that actually need browser state or event handlers.

A useful separation is:

• Server-render the page's purpose and primary content. The title, main heading, service description and important links should not depend on a button being clicked.
• Hydrate interactive elements. Menus, filters, calculators and other client-side controls can remain interactive components.
• Fetch critical data before returning the page. Do not show a permanent loading skeleton because a client-side request failed or was delayed.

The question is not whether a framework uses JavaScript. It is whether the essential page is available and understandable when the crawler processes the rendered document.

3. Check robots.txt, meta robots and X-Robots-Tag separately

These controls are related, but they do different jobs.

• robots.txt controls whether a crawler may fetch a URL.
• A noindex meta tag or X-Robots-Tag tells a crawler not to index a fetched resource.
• A canonical link suggests which URL should represent duplicate or substantially similar content.

Check the robots file:

curl -sS https://www.example.com/robots.txt 

Then inspect the response headers for X-Robots-Tag and the HTML for a robots meta tag. A page can be blocked in one place and allowed in another; check the actual URL and any relevant assets, not just the homepage.

A simple Next.js App Router app/robots.ts might look like this:

import type { MetadataRoute } from "next"; export default function robots(): MetadataRoute.Robots { return { rules: { userAgent: "*", allow: "/", disallow: ["/admin/"], }, sitemap: "https://www.example.com/sitemap.xml", }; } 

Only disallow paths that should genuinely be excluded from crawling. Do not block JavaScript or CSS resources that are needed to render important public pages.

One common mistake is using noindex as if it were a crawl blocker. It is not. A crawler must be able to fetch a page to see its noindex instruction. Conversely, blocking a URL in robots.txt can prevent a crawler from seeing the page-level directive.

Also check for an accidental deployment-wide noindex. It can be introduced through a shared layout, a staging configuration or a response header at the hosting layer. If the rule is inherited from a root layout, individual pages may look correct in their own files while still receiving the wrong directive.

4. Make the canonical URL agree with the URL you want indexed

Canonical problems are particularly easy to miss when a site has multiple hostnames, trailing-slash rules or preview deployments.

For example, these may all resolve to a page that looks identical:

https://example.com/services/it-support https://www.example.com/services/it-support https://www.example.com/services/it-support/ https://preview.example.com/services/it-support 

Choose the preferred production URL and make redirects, internal links, sitemap entries and canonical tags agree with it.

In a Next.js App Router page or server layout, metadata can be declared like this:

import type { Metadata } from "next"; export const metadata: Metadata = { metadataBase: new URL("https://www.example.com"), title: "Managed IT Support for Small Businesses in Leeds", description: "IT support, device management and practical help for small businesses in Leeds.", alternates: { canonical: "/services/it-support", }, }; 

The metadataBase setting lets Next.js resolve URL-based metadata such as a relative canonical path. Use the real production origin, not a preview or localhost address.

There is an important App Router detail: Next.js supports the metadata object and generateMetadata only in Server Components. If a page begins with "use client", move the metadata export to a server component or parent layout rather than trying to export it from the client component. See the Next.js Metadata API documentation.

After deployment, inspect the actual HTML. Do not assume the source file you edited is the only place emitting canonical tags. Check for:

• more than one canonical tag;
• a canonical that points to the homepage;
• a canonical that points to a preview hostname;
• a canonical that points to a different page with different intent;
• a canonical that changes after client-side rendering.

Google recommends declaring the canonical consistently in the original HTML. If JavaScript also changes it, the rendered canonical should not contradict the original one. See Google's canonical guidance for JavaScript sites.

5. Make internal links discoverable as links

A page can exist and still be difficult to discover if nothing links to it.

For normal navigation, use an actual anchor with a meaningful href:

import Link from "next/link"; export function ServiceNavigation() { return ( <nav aria-label="Services"> <ul> <li> <Link href="/services/it-support">IT support</Link> </li> <li> <Link href="/services/cyber-security">Cyber security</Link> </li> </ul> </nav> ); } 

Avoid making a div or button act as the only way to navigate to another page. A click handler can work for a human using a browser, but it is not a substitute for a crawlable link with an href.

Google's documentation is explicit that it extracts URLs from the href attribute of links. JavaScript-injected links can be discovered when they are valid anchor elements, but ordinary links in the HTML are easier to inspect and more robust across tools. Crawlable link guidance.

When reviewing a route, ask:

1. Is there at least one relevant internal link to it?
2. Does that link point directly to the final canonical URL?
3. Is the anchor text descriptive enough to explain the destination?
4. Is the link present in the rendered page, not hidden behind an interaction that a crawler may never trigger?

A sitemap is useful, but it should not be the only discovery path for important pages.

6. Check that the sitemap lists real, canonical, public URLs

For an App Router project, Next.js supports a sitemap.ts convention that generates /sitemap.xml. The official documentation is here: Next.js sitemap files.

Download the production sitemap and check it rather than assuming that a successful build means its entries are correct:

curl -sS https://www.example.com/sitemap.xml -o sitemap.xml 

For a sitemap index, inspect the child sitemap URLs as well. For each important entry, verify that:

• it returns the expected successful response;
• it does not redirect to a different URL unnecessarily;
• it is not blocked from crawling;
• it is not marked noindex;
• its canonical points to itself (or to the intended preferred URL);
• it contains useful, current content rather than a placeholder or deleted record.

Do not add every URL your application can technically generate. Exclude private dashboards, search-result combinations, empty category pages, test routes and duplicate parameter variations. A sitemap is a list of URLs you want considered for indexing, not a way to override noindex, canonicalisation or quality decisions.

Also avoid setting every lastmod value to the current build time. Use a real content modification date when you have one. A deployment is not necessarily a content update.

7. Test the rendered page in Search Console

Once the response, directives, canonical and links are correct, use Google's URL Inspection tool in Search Console.

Inspect the exact production URL and compare:

• the indexed result and Google's last crawl information;
• the user-declared canonical and Google's selected canonical;
• the live test result;
• the HTML and resources shown in the rendered page;
• any page availability or indexing messages.

The live test is useful because it helps separate a current fetch/rendering problem from an older indexed snapshot. It does not guarantee that Google will index the page. After correcting a meaningful issue, you can request indexing, but repeatedly requesting the same unchanged URL is not a fix for the underlying problem.

If the rendered HTML is missing the main content, investigate the data request, the component boundary and any blocked scripts or API dependencies. If the content is present but Google selects another canonical, investigate duplication and URL consistency instead. If the URL is not known to Google, improve internal discovery and check sitemap coverage.

A compact debugging order

When a Next.js page is missing from search, I would check it in this order:

This order prevents a common waste of time: rewriting content or changing keywords when the actual problem is a redirect, a noindex header or a route that returns an empty application shell.

Final thought

A successful browser load is a user-facing test. Indexability is a separate technical outcome that depends on access, rendering, directives, canonicalisation, discovery and Google's own selection systems.

For a Next.js site, the fastest route to an answer is to inspect the deployed URL from the outside first. Start with the HTTP response and HTML that actually exist in production. Then compare that evidence with the rendered page and Search Console. It is much more reliable than guessing from the component tree or repeatedly changing metadata.

AI assistance disclosure: This article was prepared with AI assistance. The publisher should test the commands against a real project, verify the framework-specific details for the installed Next.js version, and edit the article to reflect their own technical understanding before publication.

Further reading

For a related discussion of how technical accessibility, crawlability and content structure affect search visibility for smaller businesses, see SEO vs GEO: What Should a Small Business Do First? on The Super Shoppe's blog. This link is included as a relevant company-blog resource; publish this post under The Super Shoppe's DEV Organization if you intend to use DEV's company-blog backlink exception.

Technical references

• Google Search Central: Understand JavaScript SEO basics
• Google Search Central: Make links crawlable
• Google Search Central: How to specify a canonical URL
• Google Search Central: Introduction to robots.txt
• Next.js: Metadata and OG images
• Next.js: robots.txt file convention
• Next.js: sitemap.xml file convention
• Next.js: Error handling and not-found routes