Find broken links and dead sitemap URLs without installing a crawler

dev.to2026-10-13 22:060 views

Desktop crawlers are great until you need the check to run every week, on a server, for several sites, and land in a spreadsheet. Then you want something you can call from a schedule or an API.

I built two small Apify Actors for this. Disclosure: I am the developer, and both are paid per result (cents per run for a small site).

1. Broken links on the pages people actually see

Broken Link Checker crawls a site, collects every link and image on the pages it visits, and checks each URL once:

{ "startUrls": [{ "url": "https://crawler-test.com/" }], "maxPagesToCrawl": 5, "maxLinksToCheck": 150 }

On that test site, 5 pages gave 143 unique URLs and 8 broken ones in under a minute. Each row says which pages link to the broken URL and with what link text, so you know what to edit:

{ "url": "https://crawler-test.com/robots_protocol/non_200_with_noindex", "status": 404, "severity": "error", "problem": "Not found (404)", "foundOn": [{ "page": "https://crawler-test.com/", "text": "Non-200 with Noindex", "tag": "a" }] }

Details that matter in practice:

HEAD first, and every error is confirmed with a GET (many servers answer HEAD wrongly), without downloading the file.
401, 403 and 429 are warnings, not "broken": the page exists but refused a bot.
robots.txt is respected; a host that does not answer at all is reported as unreachable, not hidden.
With reportRedirects, internal links that go through a 301/302 are listed too. Updating them saves a hop.

2. Pages your sitemap promises but cannot deliver

A sitemap should list only live pages. Sitemap Extractor reads every sitemap (robots.txt, indexes, .gz) and, with checkStatus, checks each URL:

{ "startUrls": ["https://crawler-test.com/test_sitemap.xml"], "checkStatus": true, "maxUrlsPerSite": 60 }

18 URLs checked in 9 seconds. Redirects come back with their final URL, robots-blocked pages are marked, dead domains have a checkError.

Schedule it weekly with Changed since = 7 days to check only new or changed pages.

Run it from code

curl -X POST "https://api.apify.com/v2/acts/saliga~broken-link-checker/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \ -d '{"startUrls":[{"url":"https://example.com/"}],"maxPagesToCrawl":50}'