Most "broken link checker" pages you find online check exactly one URL: the one you paste. That is fine when you just edited a single article. It is useless after a migration, a CMS switch or a big cleanup, which is exactly when links break in bulk.
Desktop crawlers solve that, but they are overkill when you only want a quick answer to one question: is anything on my site pointing at a page that no longer exists?
Here is the approach I ended up with, plus the traps I hit along the way.
1. Don't crawl everything — crawl the hub pages
Broken links are not spread evenly. They cluster in navigation, footers, category pages and old articles that nobody revisits. The homepage links to most of those.
So a "good enough" whole-site check is:
- Fetch the homepage.
- Collect internal links from it.
- Open the first 9–10 of them (skip images, PDFs, archives).
- Collect every link from all of those pages.
- Check each unique link once, and remember where it was found.
On a typical small business site that is 10 pages and 200–300 unique links, which covers the menu, the footer, the main categories and the newest content.
2. Check links in parallel, not one by one
A sequential loop with an 8-second timeout per link takes minutes as soon as a few hosts are slow. In PHP, curl_multi keeps 10–15 requests in flight and finishes the same job in seconds:
$mh = curl_multi_init(); foreach (array_splice($queue, 0, 15) as $url) { $ch = curl_init($url); curl_setopt_array($ch, [ CURLOPT_NOBODY => true, // HEAD request CURLOPT_FOLLOWLOCATION => true, CURLOPT_MAXREDIRS => 5, CURLOPT_TIMEOUT => 8, CURLOPT_RETURNTRANSFER => true, ]); curl_multi_add_handle($mh, $ch); } // then loop curl_multi_exec / curl_multi_info_read, // adding a new handle each time one finishes 3. HEAD requests lie sometimes
This was the biggest source of false positives. Some servers answer HEAD with 405 Method Not Allowed, 403 or 501, even though the page opens fine in a browser. CDNs and WAFs also drop HEAD requests now and then, so you get a timeout.
The fix is simple: anything that comes back as 403/405/501 or with no status at all gets re-checked with a normal GET. Only then do you call it broken.
4. A redirect is not "OK"
If CURLOPT_FOLLOWLOCATION is on, curl reports the status of the final URL, so a link that goes through two redirects shows up as a clean 200. Check CURLINFO_REDIRECT_COUNT instead.
Redirected internal links are not errors, but they are worth fixing:
- every hop adds latency,
- chains of 3+ redirects may not be followed to the end,
- after a migration they are the first sign of links that still point at old URLs.
5. Report where the link is, not just what it is
A list of 404 URLs is half a report. What you actually need is "this broken link sits on /blog/old-post/ with the anchor Read more", so you can go and fix it. Keep the first few source pages for each link, and show the count of the rest.
What I use now
I wrapped all of this into a free tool: broken link checker for a whole website. Paste the homepage, choose Whole website, and it checks up to 10 pages and 300 links, with the anchor text and the page each link was found on. For a single chain of redirects there is a separate redirect checker.
If the site is bigger than that, a real crawl is still the right tool. But for the "did the migration break anything?" question, ten hub pages answer it surprisingly often.
What is your go-to way of catching broken links after a redesign?
