Most web data is about the present: today’s prices, today’s rankings, today’s pages. But many questions need the past. How has a competitor’s pricing page changed over three years? When did a policy page add a clause? What did a site look like before a redesign? For teams that gather web data, web archives are often the only place those answers still exist.
The largest is the Internet Archive’s Wayback Machine. This guide covers what it holds, how to query it, where its data falls short, and why access is getting harder.
What the Wayback Machine holds
On 22 October 2025, the Internet Archive celebrated “1 trillion web pages preserved”, available through the Wayback Machine. A 2026 presentation by its infrastructure team put the current pace at about 800 million pages captured per day, with more than 210 petabytes of unique data across all of the Archive’s collections.
Two points about that number. First, a “page” here is a capture, so a popular URL can appear thousands of times. Second, the Archive does not publish how many unique URLs it holds, so the trillion says more about depth on some pages than breadth across the web.
For contrast, Common Crawl’s August 2026 crawl held 2.14 billion pages, and is openly downloadable in bulk. The two answer different questions: Common Crawl is a broad monthly sample, and the Wayback Machine is a long history of specific URLs. Our guide to Common Crawl covers when that kind of dataset is enough.
How to query it
The Wayback Machine has two documented interfaces for programmatic access.
| Interface | What it returns | Good for |
|---|---|---|
| CDX Server API | One line per capture: URL key, timestamp, original URL, MIME type, status code, content digest, length | Listing and filtering captures |
| Availability API | The single closest capture to a date | Checking one URL at one time |
The CDX API does most of the work. It can filter by status code or MIME type, collapse adjacent duplicates, and page through large results; its documentation recommends pagination for bulk queries. The trick that saves the most effort is collapsing on the content digest, so you only see captures where the page actually changed.
import time
import requests
UA = "research-pipeline/1.0 (data-team@example.com)"
def captures(url, start="2020", end="2026"):
params = {
"url": url,
"output": "json",
"fl": "timestamp,original,statuscode,digest",
"filter": "statuscode:200",
"collapse": "digest", # keep only captures where the content changed
"from": start,
"to": end,
}
r = requests.get("https://web.archive.org/cdx/search/cdx",
params=params, headers={"User-Agent": UA}, timeout=60)
r.raise_for_status()
rows = r.json()
return [dict(zip(rows[0], row)) for row in rows[1:]]
def fetch(capture):
# id_ returns the archived bytes without the Wayback banner or rewritten links
url = f"https://web.archive.org/web/{capture['timestamp']}id_/{capture['original']}"
time.sleep(5) # one request at a time, with a pause
return requests.get(url, headers={"User-Agent": UA}, timeout=60)
for c in captures("example.com/pricing"):
print(c["timestamp"], c["digest"])Start from the CDX list, decide which captures you actually need, and fetch only those. A list of 2,000 captures often collapses to a few dozen distinct versions.
Where coverage falls short
An archive only holds what it captured, and capture is uneven.
- Not everything is archived. A study of URLs drawn from several sources estimated that between 35% and 90% of the web had at least one archived copy, depending on the sample, and that no more than 31.3% of URLs were archived more than once a month.
- The web keeps disappearing. Pew Research found in 2024 that 38% of pages that existed in 2013 were no longer accessible on the live web. For many of those, an archive copy is all that is left, if there is one.
- Coverage varies by country. Research has long found that archive coverage differs by country and language, and national web archives often cover their own domains better.
- JavaScript hides content. A study of archived pages found that JavaScript was responsible for 52.7% of missing embedded resources. Single-page apps, prices loaded by script, and content behind consent walls are often missing or empty.
So absence in the archive is not evidence that something never existed. Treat archive coverage as a sample, and record the sampling gaps along with the data.
Where quality falls short
Even a capture that exists may not show what a visitor saw.
Time drift inside one page. The HTML, images, scripts and stylesheets of a page are often captured at different times. One study of 82,425 archived pages across 15 archives found that only 38.7% were consistent in time, and only about one in five, 17.9%, was both consistent and complete.
Off-topic captures. A URL’s history can include parked domains, error pages and hijacked content. One study found off-topic pages in 24% of the capture histories it sampled.
Silent errors. A capture with status 200 may still be a block page, a consent wall or a soft 404. The same checks we recommend for silent failures in live scraping apply here: validate the content, not just the status code.
For analysis, that means comparing text and structured fields across versions, not screenshots, and keeping the capture timestamp with every extracted value. Our guide to change detection covers diffing versions without drowning in noise.
Why access is getting harder
Two pressures are narrowing what archives can offer.
Sites are keeping the archive out. In August 2025, Reddit began limiting the Wayback Machine’s access, saying AI companies had scraped its data from the archive. By January 2026, Nieman Lab found that 241 of 1,167 news sites across nine countries disallowed at least one of the Internet Archive’s crawlers. The New York Times said: “We are blocking the Internet Archive’s bot from accessing the Times.” The Archive’s own FAQ notes that publishers could not point to a particular AI company scraping through it, but the gaps in coverage are real either way.
The Archive is defending itself. In September 2026, the Internet Archive said the Wayback Machine had been “hit by waves of high-volume automated traffic”, that it had put protections in place, and that blocked requests receive a 429 error. Its FAQ says: “We use rate limiting, filtering, and monitoring to prevent abusive access,” and “The Wayback Machine is built for human readers.”
The Archive is a non-profit library, and it has been taken offline before by bulk traffic. Heavy users are the reason the protections exist.
How to use archives well
- Query the index, not the pages. Use the CDX API to find the captures you need, collapse by digest, and fetch the minimum.
- Go slowly and identify yourself. One request at a time, with pauses, and a descriptive User-Agent. For large projects, contact the Archive first.
- Check other archives. National libraries and other archives, reachable through the Memento standard, can fill gaps.
- Record provenance. Keep the capture timestamp, original URL and digest with every value, as our guide to provenance in the AI training pipeline describes.
- Report gaps. Say how many URLs had no captures, or no usable ones, so readers can judge the result.
Where proxies fit, and where they do not
Do not use proxies to get around the Wayback Machine’s limits. It is a library with a public mission, its protections exist because of bulk automated traffic, and rotating IPs to avoid a 429 is exactly the behaviour it is defending against. If your project needs more than polite access allows, ask the Archive.
Residential proxies are for a different job: collecting the live web as local visitors see it today. Many historical datasets start there. If you capture the pages you care about now, from the markets you care about, you will not depend on someone else’s archive later.
The bottom line
The Wayback Machine is the deepest record of the web’s past, and the CDX API makes it practical to use. But its coverage is a sample, its captures can mix moments in time, and more sites are keeping it out. Query the index, fetch little, validate what you get, record where every value came from, and treat the archive as the shared public resource it is.
Sources and references
- Internet Archive, one trillion web pages archived (22 October 2025): blog.archive.org
- Jonah Edwards, Internet Archive materials update, Designing Storage Architectures meeting (2026): digitalpreservation.gov
- Wayback CDX Server API documentation: github.com
- Wayback Machine APIs: archive.org
- RFC 7089, HTTP Framework for Time-Based Access to Resource States (Memento): rfc-editor.org
- Common Crawl, August 2026 crawl archive: commoncrawl.org
- Ainsworth et al., How much of the web is archived?: arxiv.org
- Pew Research Center, When online content disappears (17 May 2024): pewresearch.org
- AlSum et al., profiling web archive coverage: arxiv.org
- Brunelle et al., the impact of JavaScript on archivability (2016): doi.org
- Ainsworth, Nelson and Van de Sompel, temporal coherence of archived pages (2015): ws-dl.blogspot.com
- AlNoamany, Weigle and Nelson, detecting off-topic pages in web archives (2016): doi.org
- Nieman Lab, News publishers limit Internet Archive access (28 January 2026): niemanlab.org
- Engadget, Reddit restricts the Wayback Machine (August 2025): engadget.com
- Internet Archive, An update on Wayback Machine access (15 September 2026): blog.archive.org
- Internet Archive, FAQ on publishers blocking the Wayback Machine: help.archive.org