I work on a company site we moved from an old domain to a new one. The old domain redirects every path with a 301, except robots.txt, which stays reachable so crawlers can still read it.

During a check, robots.txt on the old domain answered 403 to a request with the bingbot user agent. The same URL answered 200 in a browser. It went on our to-do list as "CDN blocks Bing". Before changing anything on a domain that is still in the middle of a move, I measured it properly. The to-do item turned out to be poorly supported.

The measurement

The old domain sits behind Hostinger's CDN (server: hcdn). I sent 20 requests per user agent from a single IP, with a random query string to get past the cache and a 0.7 second pause between requests:

Chrome UA 200: 20 403: 0 bingbot UA 200: 18 403: 2 Googlebot UA 200: 18 403: 2

Two things stand out. The 403 is intermittent: the same user agent mostly gets through. And it only showed up for the crawler user agents, which also fits with an earlier run where bingbot got a 403 on one request and a 200 on the next.

Without the query string the result is even less useful, because a cached copy may answer. In one run bingbot got a cache HIT and a 200, while Googlebot got a 403 with no cache status at all. A single curl call tells you very little here.

What the real crawler reported

My requests only claimed to be crawlers. They came from an ordinary IP address, and that difference may be the whole story. So I looked at what a real crawler recorded.

The robots.txt report in Google Search Console lists all four variants of the old domain (http and https, with and without www) as "Fetched" with no issues. The dates differ: the two http variants on 2 November and 30 October, the two https variants on 14 and 13 October. All of them predate my test.

That covers Google only. I have no equivalent report for Bing on this property, so I can't say what the real Bingbot gets. One possible explanation is that the CDN sometimes rejects requests that claim to be a crawler but come from an IP outside the crawler's range. That is my inference, not something Hostinger documents in anything I read, and 2 of 20 against 0 of 20 is too small a sample to show a real difference.

How I test this now

  1. Don't treat a spoofed user agent as a test of crawler access. It tests how your stack treats an impostor.
  2. If you do test it, repeat it with a cache buster and count the results. One request can land on either side.
  3. Check the crawler's own report first: the robots.txt report in Search Console, and URL inspection in Bing Webmaster Tools if you have the property verified.
  4. For a definite answer, read the access logs and verify the client IP. Google documents how to confirm its crawler by reverse DNS, and Bing Webmaster Tools has a Verify Bingbot tool.

What we did

Nothing, for now. Turning off the CDN on a domain with a running move, to fix a problem Google's report doesn't show, would have added risk for no measured gain. The open point is Bing, which we'll check once we have data for it.

Disclosure: this post, including the measurement on 3 November 2026, was produced by an AI agent working for DripBack GmbH, with little human editing. I'm Timo Korte, and I publish it under my name. The numbers come from that test.