Add a smoke test to your deploy that fetches your key URLs and fails if any of them says "do not index". It takes about 40 lines, and it catches the most expensive small mistake in a site launch: a noindex that was right on staging and stayed on in production.
The mistake is common after a redesign. The staging site is blocked on purpose. Someone copies a config, a layout or a header rule across, and nobody looks at it again. Pages drop out of search over the next weeks, and the cause is invisible in the browser because the page looks fine.
Where "do not index" can hide
There are four places. A good check reads all four.
- The meta tag in the HTML head:
<meta name="robots" content="noindex">. It can also be addressed to one bot, such asgooglebot. - The
X-Robots-Tagresponse header. This one is easy to miss, because you never see it in the page source. It is often set in a server or CDN rule. -
robots.txt. ADisallowrule does not remove a page, but it stops crawling, and a blocked page cannot show Google anoindexor a canonical tag. - The canonical tag. A canonical that points to another address tells Google the page you are checking is a copy.
The script
This one runs on Node 18 or later, which has fetch built in. Save it as check-index.mjs.
// usage: node check-index.mjs https://example.com / /services/ /contact/ const [base, ...paths] = process.argv.slice(2); if (!base || paths.length === 0) { console.error("usage: node check-index.mjs <base-url> <path>..."); process.exit(2); } const problems = []; // 1. robots.txt: does a Disallow match any of our paths? const robots = await fetch(new URL("/robots.txt", base)).then((r) => (r.ok ? r.text() : "")); const disallows = []; let appliesToAll = false; for (const line of robots.split("\n")) { const l = line.trim(); if (/^user-agent:/i.test(l)) appliesToAll = l.split(":")[1].trim() === "*"; if (appliesToAll && /^disallow:/i.test(l)) { const rule = l.split(":")[1].trim(); if (rule) disallows.push(rule); } } for (const path of paths) { const url = new URL(path, base).href; const res = await fetch(url, { redirect: "follow" }); const html = await res.text(); if (res.status !== 200) problems.push(`${url}: status ${res.status}`); // 2. header const header = res.headers.get("x-robots-tag") || ""; if (/noindex|none/i.test(header)) problems.push(`${url}: X-Robots-Tag says "${header}"`); // 3. meta tag (robots or any named bot) const metas = [...html.matchAll(/<meta[^>]+name=["'](robots|googlebot)["'][^>]*>/gi)]; for (const m of metas) { if (/noindex|none/i.test(m[0])) problems.push(`${url}: meta tag says noindex (${m[1]})`); } // 4. canonical points somewhere else? const canon = html.match(/<link[^>]+rel=["']canonical["'][^>]*href=["']([^"']+)["']/i); if (canon) { const target = new URL(canon[1], url).href; if (target !== url) problems.push(`${url}: canonical is ${target}`); } // robots.txt const p = new URL(url).pathname; for (const rule of disallows) { if (p.startsWith(rule)) problems.push(`${url}: robots.txt disallows "${rule}"`); } } if (problems.length) { console.error("Index check failed:\n" + problems.map((p) => " - " + p).join("\n")); process.exit(1); } console.log(`Index check passed for ${paths.length} URL(s).`); Run it against production after each deploy:
node check-index.mjs https://example.com / /services/ /contact/ /blog/ In GitHub Actions, add it as a step after the deploy step. A non-zero exit fails the job, and that is the whole point.
What to put in the list of paths
Do not test every page. Test the ones that must never be blocked:
- the home page,
- one page of each template (a service page, a blog post, a product page),
- the page that earns the most money,
- the contact page.
Five to ten URLs are enough. A template-level mistake affects every page of that template, so one sample per template finds it.
False alarms you will meet
- Trailing slashes.
https://example.com/servicesmay redirect to/services/. The script follows redirects, so compare the canonical withres.urlinstead of the URL you requested if your site normalises this way. - Pages that should be noindex. A thank-you page or an internal search page is allowed to say noindex. Keep those out of the list.
- Bot-specific rules. The
robots.txtparser above only looks at the*group. If you write rules for named bots, extend it.
What this does not catch
It does not tell you whether Google has indexed the page. For that, open Search Console and use URL Inspection. The script answers a narrower question: is your own site telling Google to leave the page out? That is the cause you can fix in five minutes, so check it first.
One more habit
Keep staging blocked with a login, not with noindex. A login cannot be copied to production by accident, because production visitors would be asked for a password on day one. A meta tag can travel without anyone noticing.

