Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com

You shipped a big release last Tuesday: a new navigation, a router upgrade, maybe a URL change in one section. A week later, someone asks whether it affected SEO.

Search Console's crawl stats report gives you a sitewide graph, a few days delayed, with limited breakdowns. What you actually want to know is more specific: did Googlebot crawl any section of the site less after the deploy, and what was it getting instead?

Your access logs have the answer, with every request Googlebot made, to the second. This script compares the days before a cutoff with the days after, per section, and shows where crawling dropped and which status codes changed.

What the script does

  1. Reads Nginx or Apache combined-format logs (plain or gzipped).
  2. Keeps only requests from Googlebot's user agent.
  3. Groups URLs into sections by their first path segment (/blog/, /docs/, /products/).
  4. Counts requests per day, per section and status class, before and after a cutoff time.
  5. Prints sections whose daily crawl rate changed the most, with the status mix for each period.

The script

#!/usr/bin/env python3 """Compare Googlebot crawling per section before and after a cutoff.""" import gzip, re, sys from collections import Counter, defaultdict from datetime import datetime, timezone LINE = re.compile(r'\[(?P<ts>[^\]]+)\] "(?:GET|HEAD) (?P<path>\S+) [^"]*" (?P<status>\d{3}) .*"(?P<ua>[^"]*)"$') cutoff = datetime.fromisoformat(sys.argv[1]).astimezone(timezone.utc) files = sys.argv[2:] counts = defaultdict(Counter) # (period, section) -> Counter(status class) days = {"before": set(), "after": set()} def section(path): path = path.split("?", 1)[0] parts = [p for p in path.split("/") if p] return "/" + parts[0] + "/" if parts else "/" for f in files: opener = gzip.open if f.endswith(".gz") else open with opener(f, "rt", errors="replace") as fh: for line in fh: m = LINE.search(line.rstrip()) if not m or "Googlebot" not in m["ua"]: continue ts = datetime.strptime(m["ts"], "%d/%b/%Y:%H:%M:%S %z").astimezone(timezone.utc) period = "before" if ts < cutoff else "after" days[period].add(ts.date()) counts[(period, section(m["path"]))][m["status"][0] + "xx"] += 1 nb, na = max(len(days["before"]), 1), max(len(days["after"]), 1) rows = [] for sec in {s for _, s in counts}: b, a = counts[("before", sec)], counts[("after", sec)] rb, ra = sum(b.values()) / nb, sum(a.values()) / na rows.append((ra - rb, sec, rb, ra, b, a)) print(f"{nb} day(s) before, {na} day(s) after {cutoff:%Y-%m-%d %H:%M} UTC") print(f"{'section':<24}{'before/day':>11}{'after/day':>11} status mix before -> after") for delta, sec, rb, ra, b, a in sorted(rows)[:25]: mix = lambda c: " ".join(f"{k}:{v}" for k, v in sorted(c.items())) or "-" print(f"{sec:<24}{rb:>11.1f}{ra:>11.1f} {mix(b)} -> {mix(a)}") 

Run it with the deploy time and the logs covering a week either side:

python3 crawl_diff.py 2026-11-10T14:00+02:00 /var/log/nginx/access.log* 
7 day(s) before, 6 day(s) after 2026-11-10 12:00 UTC section before/day after/day status mix before -> after /docs/ 412.3 96.5 2xx:2801 3xx:85 -> 2xx:301 3xx:212 4xx:66 /products/ 188.0 141.2 2xx:1316 -> 2xx:802 5xx:45 /blog/ 120.4 131.0 2xx:843 -> 2xx:786 

Sections are sorted by the biggest drop first. In this example, /docs/ lost most of its crawling, and what's left is now mostly redirects and 404s, which points straight at a URL change that wasn't fully redirected. /products/ has a new block of 5xx responses worth investigating.

Reading the results

Crawl rate down, status mix unchanged. Googlebot is simply asking for fewer URLs in that section. Often that means fewer links to them: the new navigation dropped the section, or internal links were removed. Check whether the section is still linked from the main menu and the sitemap.

More 3xx after the deploy. URLs changed and redirects were added. That's fine for a while, but crawl rate on the new URLs should pick up over the following weeks. If it doesn't, check that internal links and the sitemap point at the new URLs directly, not at the redirects.

New 4xx. URLs Googlebot knew about now return 404. Find the top ones and decide: redirect, restore, or let them go. Use the full paths for that:

zcat -f /var/log/nginx/access.log* | grep Googlebot | awk '$9 ~ /^4/ {print $7}' | sort | uniq -c | sort -rn | head 

New 5xx. These are the most urgent. Repeated server errors make Googlebot slow down crawling for the whole host, not just the failing section.

Logs tell you what happened after a deploy; the same check before the next deploy is better. seotto.com watches for indexing blocks and SEO regressions as you ship, so a section that started returning redirects or errors to crawlers gets flagged before its crawl rate falls.

Caveats

  • User agents can be faked. For a quick comparison it's fine, since fake Googlebots are usually a small, steady share. For anything precise, verify IPs with reverse DNS or Google's published ranges first.
  • Crawling varies naturally. Day-to-day swings of 20 to 30% are normal. Compare at least a week on each side, and look for big changes, not small ones.
  • Weekly patterns. If possible, compare the same weekdays on both sides of the cutoff.
  • Logs behind a CDN. If a CDN serves cached responses, your origin logs only show misses. Use the CDN's logs if you can, or the comparison will under-count.

Make it routine

Run the script a week after every significant release, and keep the output with the release notes. After a few releases you'll know what "normal" looks like for your site, and an unusual drop will stand out immediately.

Related reading

Do you check crawl behaviour after releases? I'm curious whether anyone has automated this as part of a deploy pipeline.