Case study (hypothetical): the staging site that got indexed, and started outranking production

This is a hypothetical case study. The company, numbers and timeline are invented to illustrate a pattern that is genuinely common. No real site or data is described.

A support ticket arrives on a Monday: "Your search result links to a weird address and asks me for a password." Someone on the team searches for the company's product name. The second result is the real site. The first is staging.example-shop.com.

Here's how a staging environment ends up in search results, why the obvious fix often makes things worse, and what protection actually works.

The setup

Our fictional company, a small online shop, ran a staging site to preview changes before release. It was a copy of production: same code, same templates, a recent copy of the product catalogue. To keep it private, the team relied on two things:

Nobody knew the address.
A robots.txt on staging that said Disallow: /.

Neither of those is access control.

How it got found

In our scenario, nobody did anything dramatic. Over a few months:

A developer pasted a staging link into a public issue tracker while reporting a bug in an open-source library.
A staging page was shared in a chat tool that generated a link preview, and the link ended up in a public channel archive.
The staging site's sitemap, copied from production, listed every product URL on the staging domain.

Search engines discover URLs from links anywhere on the web. Once they'd found one staging URL, the sitemap handed them the rest.

Why robots.txt didn't help

This is the part that surprises people. robots.txt asks crawlers not to fetch pages. It doesn't stop a URL from being indexed based on links pointing to it. A blocked URL can still appear in results, usually with little or no description, because the search engine knows the URL exists but hasn't been allowed to read it.

Worse, a blocked URL can't be crawled, so the search engine can never see a noindex tag on it. The block hides the very instruction that would remove the page.

Why staging started winning

In our hypothetical, staging pages began to appear above production for some product searches. Plausible reasons:

Duplicate content across domains. Two copies of every product page, and search engines had to pick one. Without clear signals, they sometimes picked staging.
Canonical tags pointing at themselves. The templates generated a canonical URL from the current host, so staging pages declared staging as the canonical version.
Staging was sometimes faster. No real traffic, no third-party scripts in some builds.

The result: customers clicked a search result, hit a password prompt or a half-finished page, and left.

The cleanup (in the right order)

The team's first instinct was to add a password to staging immediately. That's the right end state, but the order matters.

Step 1: Make staging pages say noindex, and let them be crawled. Temporarily remove the Disallow: / from staging's robots.txt and send an X-Robots-Tag: noindex header on every staging response. Now crawlers can fetch the pages, see the noindex, and drop them.

# staging server block add_header X-Robots-Tag "noindex, nofollow" always;

Step 2: Fix canonicals. Canonical URLs should be built from a configured production base URL, not the request host. Staging pages now point at their production equivalents.

Step 3: Remove the staging sitemap, or make it empty. Don't keep advertising the URLs.

Step 4: Verify the staging domain in the search engine's webmaster tools and use the removals tool to hide the worst URLs quickly while the noindex takes effect. Removal requests are temporary; the noindex is what makes it permanent.

Step 5: Watch. Search the staging domain with a site: query every few days until the results disappear.

Step 6: Only then add authentication. Once the staging URLs are gone from the index, put staging behind a password or an IP allowlist. From then on, crawlers get a 401 and nothing new can be indexed.

In our scenario, if the team had added the password first, crawlers would have hit 401s on URLs they already knew about. Those generally drop out eventually too, but more slowly and less predictably than with an explicit noindex.

The protection that works

After the cleanup, the team set rules so it couldn't happen again:

Authentication on every non-production environment. HTTP basic auth, a single-sign-on proxy or an IP allowlist. This is the only real protection. Everything else is a backup.
A noindex header as a second layer, set by the environment, not by the application code that might be copied to production.
Canonicals and sitemaps from configuration. The production base URL is a setting. It's never inferred from the request.
A check in CI that fails if a production build is missing the right canonical host, or if staging responds without the noindex header.

curl -sI https://staging.example-shop.com/ | grep -i x-robots-tag || echo "MISSING noindex on staging"

A deploy check for the opposite mistake. The scariest version of this bug isn't staging getting indexed; it's the noindex header leaking into production. Check production on every deploy too:

curl -sI https://www.example-shop.com/ | grep -i "x-robots-tag: .*noindex" && echo "PRODUCTION IS NOINDEXED"

Lessons

Obscurity isn't privacy. URLs leak through issue trackers, link previews, browser extensions and sitemaps.
robots.txt blocks crawling, not indexing. To remove a page, it has to be crawlable and say noindex.
Derive canonical URLs from configuration, never from the incoming request.
Authentication is the real fix. Add it after cleanup, and to every new environment from day one.
Test both directions: staging must say noindex, production must not.

Preview deployments, the per-branch URLs many hosting platforms create automatically, deserve the same treatment: they're staging sites too, just more of them.

If you want a second pair of eyes on how your environments look to search engines, that's the kind of thing we check at seotto.com.