Recover deleted pages and old URLs with the Wayback Machine, in bulk

Three situations where I have needed the Wayback Machine, and where its website is the wrong tool:

1. After a site migration, traffic dropped. Which old URLs used to exist, and which of them now 404 without a redirect?
2. An old blog post has 40 outbound links, and a third of them are dead. Each needs an archive link instead.
3. An expired domain is for sale with "20 years of history". Was it a real site for 20 years, or a parked redirect for the last ten?

The Internet Archive has a public index API (the CDX server) that answers all of these. It is free, generous and also slow and easy to overload: heavily archived pages can take 30–60 seconds per query, and it starts refusing at about 24 queries a minute. So I wrapped it in an Apify Actor, Wayback Machine Scraper, that queues politely, retries, de-duplicates, and returns clean rows. Here is how each job looks.

Job 1: every URL a site ever had

Mode urls lists every page the Wayback Machine has captured for a domain, a host or a path, with the first capture date:

{ "urls": ["https://blog.example.com/"], "mode": "urls", "matchType": "prefix", "fromDate": "2020", "maxResultsPerInput": 5000 } 

A real row for python.org:

{ "type": "url", "input": "python.org", "url": "http://www.python.org/1.5/", "path": "/1.5/", "firstCapturedAt": "1998-01-19T01:50:21Z", "statusCode": 200, "mimeType": "text/html", "firstSnapshotUrl": "https://web.archive.org/web/19980119015021/http://www.python.org:80/1.5/", "latestSnapshotUrl": "https://web.archive.org/web/http://www.python.org/1.5/", "charged": true } 

By default you get only successful HTML pages, and http://example.com:80/a/ and https://www.example.com/a count once. That is what you want for a redirect map: export the path column, diff it against your current sitemap, and every path that is in the archive but not on the new site is a candidate for a 301. Add urlPattern (a regex) to focus on, say, /blog/ or /products/.

Job 2: an archive link for every dead link

Mode closest returns the snapshot nearest to a date (or the latest one) for each URL you paste:

{ "urls": ["https://www.nasa.gov/missions/", "https://example.com/old-page"], "mode": "closest" } 

With targetDate and includeContent, you also get the archived page as Markdown, which is handy when you want to quote what a page said rather than just link to it:

{ "type": "closest", "input": "https://www.nasa.gov/missions/", "targetDate": "2015-06-01T00:00:00Z", "capturedAt": "2015-06-12T18:50:45Z", "statusCode": 200, "snapshotUrl": "https://web.archive.org/web/20150612185045/http://www.nasa.gov:80/missions/", "daysFromTarget": 12, "contentTitle": "NASA Missions A-Z | NASA", "contentFormat": "markdown", "charged": true } 

A URL that was never archived gets a free row (errorType: "not_found") instead of a fake answer.

This pairs naturally with a link checker: run Broken Link Checker on your site, feed its dead links into closest mode, and you have a replacement-link sheet for your CMS.

Job 3: every version of a page, with the text

Mode snapshots lists the archived versions of one page. With the default collapse: "changes", you get one row per content change, not one per crawl, which removes most of the noise:

{ "urls": ["stripe.com/pricing"], "mode": "snapshots", "collapse": "changes", "includeContent": true, "maxContentPages": 20 } 

{ "type": "snapshot", "input": "stripe.com/pricing", "index": 2, "capturedAt": "2020-01-04T16:56:45Z", "statusCode": 200, "digest": "GMSUHBJQMSMCD7CWRFAY4X2OPZGY5BJW", "lengthBytes": 69301, "snapshotUrl": "https://web.archive.org/web/20200104165645/https://stripe.com/pricing", "rawUrl": "https://web.archive.org/web/20200104165645id_/https://stripe.com/pricing", "contentChanged": true, "daysSincePrevious": 243 } 

digest is the capture's content fingerprint (same digest = same content), and rawUrl is the file exactly as captured, without the Wayback toolbar. With includeContent, each version comes as Markdown, ready to diff or to hand to an LLM: "how did this pricing page change between 2019 and 2022?"

Job 4: is this expired domain what it claims?

Mode history gives one summary row per domain. This is pets.com, a real result:

{ "type": "history", "input": "pets.com", "firstCapturedAt": "1996-12-21T00:37:03Z", "lastCapturedAt": "2026-09-20T01:06:47Z", "lastOkMonth": "2016-04", "archiveAgeYears": 29.7, "totalCaptures": 2391, "monthsWithCaptures": 268, "monthsWithOkPages": 40, "monthsRedirectOnly": 228, "longestGapMonths": 19, "distinctVersions": 117, "redirectShare": 0.494, "calendarUrl": "https://web.archive.org/web/*/pets.com" } 

"Archived for almost 30 years" sounds impressive. But the last month with a real page (HTTP 200) was April 2016, and 228 of the 268 months with captures were redirect-only. If you were evaluating this as an aged domain for SEO, that is the line that matters, and it is exactly the line a marketplace listing will not show you.

What it costs

No start fee. So 1,000 archived URLs of a site cost $1, and a history check of 100 domains costs $0.30. Inputs that were never archived, invalid lines, duplicates, filtered-out captures and content that could not be downloaded are free.

A real example from my tests: two pages, closest snapshot to June 2015 with their Markdown, plus one never-archived domain = 2 captures + 2 page contents = $0.008.

Limits worth knowing

• The Wayback Machine is a shared free service. The Actor defaults to one index query every 2.5 seconds and one page download every 5 seconds, and backs off when the archive is busy. Big jobs take time; that is the price of not hammering it.
• Only what was archived. Pages that were never crawled, or were excluded or removed at the owner's request, are not available.
• JavaScript-heavy pages may have little text in their archived HTML.
• For live change alerts on a page today, use a change monitor; this tool is for the past.

Terms and responsible use

The Actor reads only the Internet Archive's public endpoints (the CDX index, the availability API and snapshot playback), at a gentle rate, without logging in. Archived pages remain their owners' work, so check copyright before republishing them. I am not affiliated with the Internet Archive; if you rely on it, consider donating.

Actor: https://apify.com/tidytools/wayback-machine-scraper

Disclosure: I built this Actor and earn from its usage on Apify. The sites above are public examples from my tests in September 2026 and have no connection to me.