Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com
Search Console's Pages report has a reason that looks simple: Blocked due to access forbidden (403). Googlebot requested the URL and got HTTP 403 Forbidden, so the page can't be indexed.
The confusing part is that the page usually loads perfectly in your browser. The 403 is aimed at Googlebot specifically, or at requests that look like it, or at requests from where Google crawls. Something between Google and your content decided this visitor wasn't welcome.
Here's how to find which layer it was and how to fix it safely.
Step 1: confirm it's real and current
Pick a few example URLs from the report and run URL Inspection, then Test live URL. If the live test shows a 403, the problem is happening now. If it shows 200, the 403 was temporary (a rule that's since been changed, a brief rate limit) and you can click "Validate fix" once you're sure the cause is gone.
Also check: are these pages you actually want indexed? If they're admin pages, account pages or staging URLs that correctly return 403, the fix is to stop linking them and remove them from the sitemap, not to unblock them.
Step 2: find which layer returned the 403
A 403 can come from several places. The response itself usually tells you which:
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/page Look at the headers and body:
- CDN or WAF block pages have recognisable headers (
server: cloudflarewith acf-ray,x-amz-cf-idwith an AWS WAF block,x-sucuri-id,x-iinfoand so on) and a branded HTML page. - Your web server returns its default 403 page (
nginxorApachein theserverheader). - Your app returns its own error template.
A curl request with a Googlebot user agent from your own IP isn't the same as a real Googlebot request. Many bot rules check that "Googlebot" requests come from Google's IP ranges, and correctly block your fake one. That's fine; you're looking for which layer answers, not reproducing Google exactly. The URL Inspection live test is the real thing.
A 403 for Googlebot is the kind of deploy regression that should page you. seotto.com watches for indexing blocks, including access-forbidden responses that only crawlers see.
Step 3: the usual causes
Bot protection or WAF rules
The most common cause. A bot-management feature, a "block suspected bots" rule, a rate-limit rule or a custom firewall expression catches Googlebot. It often happens right after someone turns on a stricter security setting, or adds a rule to stop a scraper that happened to use a Googlebot-like user agent.
Fix: check the WAF or CDN's event log, filtered by the user agent containing "Googlebot" and the time of the failed crawls. Most major CDNs have a built-in "verified bots" category that identifies real search engine crawlers by IP and reverse DNS. Allow that category through before your blocking rules. Never allow by user agent string alone: anyone can send "Googlebot".
Rate limiting
Rate limits that count requests per IP will eventually catch a crawler that requests many pages from a small set of IPs. Depending on configuration, the limiter returns 403 or 429. Google treats 429 as "slow down" and comes back, while a 403 reads as "you're not allowed here", so if you rate limit crawlers at all, return 429 (or 503) and exempt verified search crawlers from limits designed for abusive clients.
Geo-blocking
If your site blocks countries, remember that Googlebot mostly crawls from US IP addresses. A site that only allows its home country, or blocks the US for licensing reasons, will block Googlebot. Exempt verified crawlers from geo rules, or serve crawlers the same content as allowed visitors.
Server-level rules
Old .htaccess or Nginx rules sometimes block user agents by pattern (deny lists of "bad bots" copied from a forum years ago, which often include bot as a substring). Search your server config for user agent matches, and for deny or return 403 rules that could match crawler requests.
File permissions
If only certain paths return 403 for everyone (including you, when you test from a fresh browser), the problem may be file system permissions or a missing index file with directory listing disabled. Nginx returns 403 when it can't read the file or there's no index document; the error log says permission denied or directory index of ... is forbidden.
Authentication and paywalls
Pages behind a login return 401 or 403 to Googlebot, which can't log in. For paywalled content you want indexed, the usual approach is to serve the full content to verified Googlebot and mark it up with structured data for paywalled content, so it's not treated as cloaking.
Step 4: verify the fix with real crawler traffic
After changing a rule:
- Run URL Inspection's live test on several affected URLs. They should return 200.
- Check your access or CDN logs over the next day for verified Googlebot requests to those paths, and confirm they get 200.
- Click "Validate fix" in the report. Validation can take a couple of weeks for larger numbers of URLs.
Keep an eye on the Crawl Stats report too. A sudden rise in 403 responses after a security change is a clear signal, and it's worth checking after every WAF rule change.
Security changes that accidentally lock out search crawlers are a recurring theme in the issues we see at seotto.com, largely because the site looks fine to everyone who checks it in a browser.
Related reading
- What your 500s, 503s and 429s tell search crawlers (and how to take a site down for maintenance without hurting SEO)
- Postmortem (hypothetical): our robots.txt returned 503 for nine days, and crawling nearly stopped
- How to tell real Googlebot from fakes in your access logs (reverse DNS and IP ranges, in Python)
- Should every URL end with a trailing slash? Let's argue about it
What caused the 403s on your site: a bot rule, a geo block, or something nobody could explain? And did anyone find out from Search Console before traffic dropped?

