Common Crawl: When a Free Crawl Is Enough, and When You Need Your Own

For a lot of web data work, the cheapest crawl is the one somebody else already ran. Common Crawl publishes a large slice of the public web every month, free of charge, and it underpins many of the text datasets used to train AI models. But it was built for breadth, not for your particular list of sources, and knowing where it stops saves teams weeks.

This guide covers what Common Crawl contains, what it does not, how to check whether it covers the pages you need, and when running your own crawl is worth it.

What you get

Common Crawl is a US non-profit that describes its goal as providing a copy of the internet to researchers, companies and individuals at no cost. The facts for its latest crawl, CC-MAIN-2026-43, captured between 5 and 18 October 2026:

Pages	2.16 billion
Registered domains	34.3 million
Hosts	41.4 million
URLs not seen in any earlier crawl	830.1 million
Uncompressed content	371.19 TiB
Cadence	About one crawl a month
Formats	WARC (raw responses), WAT (metadata), WET (plain text)
Access	Public AWS bucket in us-east-1, HTTPS downloads, URL indexes

The language mix is broad but skewed: English made up 42.6% of detected HTML pages in that crawl, followed by Russian at 6.3% and German at 5.8%.

What it leaves out

These limits come from Common Crawl’s own documentation, and they decide most build or reuse choices.

• It is a sample. Common Crawl says it does not generally archive an entire website but “a randomly selected subset of it”. Do not expect every product page or every article.
• No JavaScript. Its FAQ says JavaScript is not executed and cookies are not used, so content that only appears after scripts run is missing.
• No logins or paywalls. It states that it does not go behind paywalls or log in to any websites.
• It respects refusals. CCBot checks robots.txt first and obeys crawl-delay. In the October crawl, 4.85% of fetches were denied by robots.txt or a 403, and 18.47% were skipped. Our analysis of how much of the web sits behind a bot wall puts that figure in context.
• One vantage point. It crawls from a small, published set of addresses, so you get one view of each page. Pages that differ by country, language or device appear in only one version.
• It is a snapshot. A page appears as it was on the capture date, which may be weeks old.

Check coverage before you decide

The quickest test is to ask the index whether your sources are there. The script below finds the latest crawl and asks for captures of a URL pattern. It sends one request at a time from your own connection, as Common Crawl asks; for bulk lookups, use the Parquet URL index on AWS instead. It runs on Python 3.9 and later with requests.

from __future__ import annotations

import json

import requests

INDEX = "https://index.commoncrawl.org"
HEADERS = {"User-Agent": "coverage-check/1.0 (contact: data-team@example.com)"}


def latest_crawl():
    """The newest crawl ID, from Common Crawl's own list."""
    return requests.get(f"{INDEX}/collinfo.json", headers=HEADERS, timeout=60).json()[0]["id"]


def captures(crawl, url_pattern, limit=1000):
    """Captures of matching URLs in one crawl. One request, no proxies: the index asks for that."""
    r = requests.get(f"{INDEX}/{crawl}-index", headers=HEADERS, timeout=120,
                     params={"url": url_pattern, "output": "json", "limit": limit})
    if r.status_code == 404:  # no captures at all
        return []
    r.raise_for_status()
    return [json.loads(line) for line in r.text.splitlines()]


if __name__ == "__main__":
    crawl = latest_crawl()
    rows = captures(crawl, "en.wikipedia.org/wiki/*")
    statuses = {}
    for row in rows:
        statuses[row["status"]] = statuses.get(row["status"], 0) + 1
    print(crawl, "captures returned:", len(rows), "by status:", dict(sorted(statuses.items())))
    print("first:", rows[0]["url"], rows[0]["timestamp"]) if rows else print("none")

Output on 6 November 2026, for a pattern on English Wikipedia, capped at 1,000 results:

CC-MAIN-2026-43 captures returned: 1000 by status: {'200': 949, '301': 15, '404': 30, '429': 6}
first: https://en.wikipedia.org/wiki/!!! 20261017084408

Two things show up at once. Coverage of a big site is real but partial, so count what you get for your own patterns. And captures carry the status Common Crawl received: here 30 were 404s and 6 were 429s, so a capture is not always content. Filter on status before you use anything.

When the free crawl is enough

• Training and research on broad text. Language modelling, classifiers and web-scale statistics need breadth more than any particular page. This is what Common Crawl does best, and our guide to building training datasets from the open web covers the filtering and deduplication it still needs.
• Discovery. Finding which domains mention a topic, mapping links or estimating a site’s size is cheap with the URL index.
• Historical comparisons. Monthly snapshots going back years are hard to recreate yourself.

When you need your own crawl

• You need specific pages, completely. A sample will miss some of them, and you will not know which.
• You need it fresh. Prices, stock and rankings change daily. A monthly snapshot is history.
• The content needs JavaScript. Many modern pages only render their data in the browser.
• You need a local view. Prices, search results and ads differ by country. A crawl from one vantage point gives one answer. Collecting each market as its visitors see it is what residential proxies are for.
• You need to prove where it came from. A crawl you run yourself lets you record exactly how, when and from where each page was fetched.

Many teams combine both: Common Crawl for breadth and discovery, their own crawler for the sources that matter most.

Terms worth reading

Common Crawl keeps its own terms light, but they matter. Its terms of use say the crawled content may be subject to the terms of the sites it came from, that users agree to respect copyright and other rights, and they recommend legal advice before commercial use. The user also takes on liability for claims arising from using the content to develop, train or deploy AI systems. Since 2025, Common Crawl has also published a register of legal opt-out requests and removes affected URLs from later crawls and its tools. Reusing the data does not take you outside these questions, so keep records of what you used.

The bottom line

Common Crawl is the right starting point when you need breadth, history or a cheap first look. It stops being enough when you need particular pages, fresh data, JavaScript content or a local view. Check your coverage first with the index, filter captures by status, read the terms, and crawl yourself only where the free sample falls short.

Sources and references

• Common Crawl, FAQ: commoncrawl.org/faq
• Common Crawl, October 2026 crawl archive: commoncrawl.org
• Common Crawl, Terms of Use (last updated 7 March 2024): commoncrawl.org/terms-of-use
• Common Crawl, Get started: commoncrawl.org/get-started
• Common Crawl, URL index: commoncrawl.org/url-index
• Common Crawl statistics, crawl size, languages and crawler metrics: commoncrawl.github.io
• Common Crawl, Opt-out registry (17 September 2025): commoncrawl.org
• Common Crawl, Setting the record straight (4 November 2025): commoncrawl.org