Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com
Every blog, docs site and content-heavy app ends up with archive pages: /tags/kubernetes/, /category/tutorials/, /topics/payments/page/3/. Content management systems create them automatically, often hundreds of them.
And there's a long-running argument about what to do with them in search. One camp adds noindex to every tag and category page as a matter of hygiene. The other says those pages are some of the most useful on the site and should be indexed like anything else.
Let's argue about it.
The case for noindex
Most tag pages are thin. A tag used on one or two posts produces a page with a heading and two excerpts. There's nothing on it that isn't better on the posts themselves. Multiply by every tag someone made up once (#thoughts, #misc, #2021) and you have hundreds of low-value pages.
They duplicate each other. /tags/react/, /tags/reactjs/ and /category/frontend/ may list almost the same posts. Paginated archives add more near-duplicates: page 2 of one tag looks a lot like page 2 of another.
They compete with real pages. A tag page called "Kubernetes" can rank, or try to rank, for "kubernetes" ahead of your actual guide to Kubernetes, which is the page you want people to land on.
They use crawl attention. On big sites, thousands of archive pages and their paginated variants can soak up a lot of crawling, especially when combined with sort orders and filters.
The case for indexing them
They're hubs. A well-built category page links to every post on a topic. That's exactly the kind of internal linking that helps crawlers discover and understand content. And noindex pages tend to get crawled less over time, which can mean the links on them get followed less too.
They match broad searches. Someone searching "kubernetes tutorials" is better served by a page listing your 15 Kubernetes tutorials than by any single one. Good category pages are genuine landing pages.
Noindex doesn't fix the underlying mess. If you have 400 tags, the problem is the 400 tags, not their indexing status. Noindexing them hides the problem from search while leaving it for users.
It's easy to overdo. Teams that add noindex to "all archive templates" sometimes catch pages they didn't mean to, like a docs section index or the main blog listing.
Where I've landed
I think the answer is "it depends on the page", and a rule that's simple to automate:
- Categories (a small, curated set) are indexed, and treated as real pages. Give each one an introduction that explains what the section covers, link to the best posts first rather than only newest-first, and keep them in the main navigation. A category page should be worth landing on.
- Tags are indexed only above a threshold. A tag with fewer than, say, five posts gets
noindex, follow(crawlers can still follow the links, though as the page drops out of crawling that matters less). Tags above the threshold get the same treatment as categories: a description and curated ordering. - Merge duplicates rather than noindexing them.
reactandreactjsshould become one tag, with a redirect from the other. That's better than two noindexed pages. - Paginated pages stay indexable but plain. Self-referencing canonicals on each page (page 2 canonical to page 2, not to page 1), real links between pages, and no
noindexon page 2 and beyond, which would hide older posts from discovery. - Never block archive pages in robots.txt if you also want
noindexto work. A crawler that can't fetch the page can't see thenoindextag, and the URL can still end up indexed from links.
Archive templates are where a single template change can noindex hundreds of pages at once. seotto.com checks for indexing blocks and deploy regressions, so a
noindexthat lands on the wrong template gets caught before those pages drop out of search.
The parts I'm still unsure about
- Where the threshold is. Five posts? Ten? It probably depends on how much each post's excerpt adds. I've seen good arguments for "it should have at least as much unique text as an average post".
- Date archives.
/2024/03/pages almost never match a search. I'd noindex them or remove them, but some people use them for navigation and like them. - Author pages. On multi-author sites they matter for trust and for people searching an author's name. On single-author sites they duplicate the blog index.
- Faceted category pages in e-commerce. "Red shoes, size 42" is a very different problem from blog tags, with its own trade-offs. I'd keep that argument separate.
One more thing: noindex is slow to undo
Whatever you decide, decide it once. A page that's been noindexed for months gets crawled less and less often, so when you remove the tag, it can take weeks for Google to come back, notice, and index it again. Flip-flopping between "index tags" and "noindex tags" every few months is the worst of both worlds: you pay the recovery cost each time and never see a stable result.
A quick audit
Export every tag and category URL with its post count, then check three things:
- How many have fewer than five posts? Those are candidates for merging or
noindex. - How many pairs list mostly the same posts? Candidates for merging.
- Do any get search traffic today? Search Console's performance report, filtered by page, tells you. Don't noindex a tag page that already brings in visitors without a very good reason.
The answer is rarely "all" or "none". It's usually "about 20 of these are good pages, and the other 300 shouldn't exist".
Related reading
- How to fix 'Couldn't fetch' and 'Sitemap could not be read' in Google Search Console (a developer's checklist)
- Should every page have a meta description? Let's argue about it
- Your site search creates infinite URLs: an SEO checklist for internal search result pages
- There is no duplicate content penalty (but duplicate content still costs you)
What do you do with tag pages on your site? And has noindexing (or un-noindexing) them ever made a visible difference to your traffic?
