How to Find Every Archived URL on a Domain (Full Website History)

Looking up a single page answers “what did this page say?”. Looking up a whole domain answers bigger questions: what pages did this site ever have, which ones are gone, where did that old PDF live, and what needs a redirect after the migration? The Wayback Machine can answer all of them — if you know how to ask.

Why you'd want a full URL list

• Site migrations: find old URLs that now 404 and need redirects, including ones your current CMS no longer knows about.
• Content recovery: locate articles, PDFs and images from an older version of your own site.
• Expired domains: see what a domain hosted before you buy it or follow its backlinks.
• Competitive research: map a competitor's old site structure, product lines and landing pages.
• Investigations: find files and subdomains that were once public.

Method 1: The Wayback Machine's URL search

Type example.com/* into the Wayback Machine search box (or open web.archive.org/web/*/example.com/*). You get a filterable table of captured URLs under that host with their first and last capture dates. It's quick for small sites and browsable by hand, but it doesn't export and it gets slow for large hosts.

Method 2: The CDX API

The CDX server is the Wayback Machine's index. One request returns every capture that matches a pattern, and parameters let you collapse duplicates and filter by status or type.

# One row per unique URL under the domain and its subdomains
curl "https://web.archive.org/cdx/search/cdx?url=example.com&matchType=domain&collapse=urlkey&fl=original,timestamp,statuscode,mimetype&output=json&limit=10000"

# Only HTML pages that returned 200
...&filter=statuscode:200&filter=mimetype:text/html

# Only one section of the site
curl "https://web.archive.org/cdx/search/cdx?url=example.com/blog/&matchType=prefix&collapse=urlkey"

Method 3: Domain history in Web Archive Viewer

The free Domain history tool opens whole-site views on six archives at once, and checks the www, http and home-page variants that often hold different captures. It's the fastest way to see whether a site has any history worth digging into.

For the full list, Full URL inventory (Pro) does the CDX work for you and returns a clean table of every archived URL — up to 10,000 rows — for the host and its subdomains, this host only, or everything under a path:

• Last capture date, archived status and type for each URL.
• Facets for hosts, sections and file types, so you can jump straight to /blog/, the PDFs or a forgotten subdomain.
• Filters for pages, files, redirects and errors.
• CSV export for your migration spreadsheet.
• Check which are live — sends the list to Bulk lookup, which checks the live HTTP status of each URL, so you instantly see which old pages now 404.

A migration checklist using archived URLs

1. 1

   Export the old URL list

   Inventory the domain with collapse=urlkey, filtered to HTML pages and documents that returned 200.

2. 2

   Check what is live now

   Run the list through a live status check. Everything returning 404 or 410 is a candidate for a redirect.

3. 3

   Prioritise

   Old URLs with backlinks, traffic history or many captures (a sign of popularity) come first.

4. 4

   Map and redirect

   Point each dead URL to its closest current equivalent with a 301. Where there is none, consider restoring the content from the archive.

5. 5

   Verify

   Re-run the status check on the old list. Everything should now be 200 or a single 301 hop.

FAQ