I run a site with 6,100 URLs in its sitemap. Google downloaded that sitemap three days ago, no warnings, no errors. Today I inspected twenty randomly chosen pages from it, one API call each:
3 Submitted and indexed 11 Crawled - currently not indexed 6 URL is unknown to Google Fifteen per cent. And the six "unknown" ones are the interesting part: they are listed in a sitemap Google has, and Google has never fetched them at all.
What Search Console shows, and what it hides
The performance numbers for the last 28 days: 85 impressions, 5 clicks. Sixty of those 85 impressions are the homepage. The top queries are not tool queries:
software & it services 3 impressions position 5.7 software services 3 position 2.0 software house near me for freshers 2 position 1.0 That is a calculator site ranking for "software house near me". The pages that would answer a real query — the converters, the calculators — are not in the index to be shown.
Ninety days, by week, tells the rest:
| week | impressions | clicks |
|---|---|---|
| W31 | 13,683 | 24 |
| W32 | 24,082 | 39 |
| W33 | 68,834 | 35 |
| W34 | 27,317 | 23 |
| W35 | 191 | 3 |
| W36–W40 | 31, 32, 18, 21, 12 | ~1 each |
The cliff between W34 and W35 is where a compromise was cleaned up. Note the clicks column during the peak: 68,834 impressions produced 35 clicks, a CTR of 0.05%. Those impressions were never real demand — they were the attacker's invented URLs appearing in results. Losing them cost almost nothing. What has not come back is everything else.
The log the host does not give you
Search Console is two days behind and rounds everything into weekly shapes, so last week I gave the site its own crawler log: one line per crawler request — time, crawler, the status it was served, path, IP — and nothing at all for human visitors. Ten days in:
| day | Googlebot | bingbot | of which 410 |
|---|---|---|---|
| 29 Sep | 45 | 1,077 | 982 |
| 30 Sep | 66 | 4,114 | 3,914 |
| 1 Oct | 55 | 2,660 | 2,487 |
| 2 Oct | 22 | 2,903 | 2,708 |
| 3 Oct | 41 | 3,832 | 3,616 |
| 4 Oct | 19 | 3,967 | 3,776 |
| 5 Oct | 22 | 4,255 | 4,056 |
| 6 Oct | 22 | 4,976 | 4,795 |
| 7 Oct | 40 | 4,527 | 4,389 |
Two numbers to take from that table.
Googlebot fetches about 37 pages a day. In ten days it fetched 221 distinct real URLs, all 200s. Against a 6,100-URL sitemap, a single complete pass at that rate takes about five months. That alone explains the six "unknown to Google" pages in my sample of twenty: nothing is wrong with them, Google simply has not got there, and at this rate it will not get there soon.
bingbot fetches about 4,000 dead URLs a day, and that number is going up. Those are the URLs a cloaker invented during the compromise — a word, a slash, a word with digits between its letters — about 291 of them handed to every crawler that asked for a page, for six days.
I was wrong about the fix
Four days into this log I switched those dead URLs from 404 to 410 Gone, which tells a crawler to drop the URL rather than check again later. I then wrote that the daily count was "pointing the right way": 3,914 → 2,487 → 1,653.
It was not. Those were two partial days at either end of a four-day sample. Ten days later the sequence reads 3,914, 2,487, 2,708, 3,616, 3,776, 4,056, 4,795, 4,389 — flat to rising. A 410 does tell Bing to drop that URL, but it does nothing to the rate at which Bing works through a queue of nine thousand others, and I had read a trend into a sample too short to hold one. I have left the original numbers in that post with a correction under them, because deleting them would make my reasoning look better than it was.
What the two instruments say together
- Google is crawling — steadily, slowly, only real pages, every one a 200. There is no technical blockage: no 403s, no 404 storm, no cloaking, a clean sitemap fetch three days ago.
- Google is not indexing what it crawls: 11 of 20 sampled pages are "crawled, currently not indexed", most last crawled in August.
- Bing is crawling thirty times more than Google and spending 94% of it on URLs that have been gone for a month.
"Crawled — currently not indexed" is a judgement, not a fault. There is no header to add and no file to fix; it means Google fetched the page and decided it was not worth a slot. For a page that converts hectares to acres, competing with a hundred thousand identical pages, that judgement is not unreasonable, which is the uncomfortable part.
So the work is not technical any more. It is the slower kind: fewer, better pages, and reasons for anyone to link to them.
If you want the same two numbers for your own site
- Measure indexation, do not estimate it.
site:queries in a browser are a lossy approximation. The URL Inspection API gives you a verdict per URL, and twenty of them is a sample that will tell you something real in ten minutes. - Know how big your sitemap actually is before you argue about crawl budget. Mine says 6,100 across 54 child sitemaps, which is not a number I would have guessed; a sitemap generator that walks your own site will tell you what you are really asking Google to read.
- Look at the result you are competing for. Before writing another near-duplicate page, put the title and description into a SERP preview and ask whether that line would get clicked next to nine others exactly like it. If the honest answer is no, indexing is not your bottleneck.
