A crawl is not a corpus. Fetching billions of pages is an infrastructure problem, and it is covered in collecting web data for AI and LLM training. Turning those pages into training data that improves a model is a different problem, with its own stages, and most of the quality of the final dataset is decided there.
This guide walks through those stages in order: deciding what to collect, respecting opt-outs, extracting text, filtering, deduplicating, removing evaluation contamination, and documenting the result. The general scraped-dataset pipeline, at smaller scale, is in how to build a dataset with web scraping.
Start from the mixture, not the crawler
Decide what the model needs before deciding what to crawl. A training corpus is a mixture: shares of languages, domains, formats and time periods, expressed as a token budget for each.
That target decides everything downstream. It tells you which regions to collect from, how much crawl budget each source deserves, and when a domain has contributed enough. Without it, a crawl drifts toward whatever is easiest to fetch, which is usually large, English-language, heavily linked sites, and the model inherits that imbalance. The sampling side of that problem is covered in your residential proxy pool is a sample, not the internet.
The crawl frontier
The frontier is the queue of URLs waiting to be fetched, and how you manage it decides which parts of the web end up in the corpus.
Seeds. Start from sources that match the mixture: curated domain lists per language and topic, sitemaps, and links from high-quality pages.
Prioritisation. Rank URLs by expected value: a quality prior for the source, novelty relative to what you already hold, and freshness where recency matters.
Canonicalisation. Normalise URLs before queueing, removing tracking parameters, session identifiers and duplicate paths, so the same page is not fetched many times under different addresses.
Politeness budgets. Cap concurrency and request rate per host, not just globally. A frontier that hammers a small site is both irresponsible and self-defeating, since it gets blocked. The mechanics are in rate limiting and request throttling.
Recrawl policy. Decide how often each source is revisited, based on how often it changes and how much fresh content the mixture needs.
Respect opt-outs at fetch time
Opt-out signals have to be checked when a page is fetched and recorded alongside it, because they change over time and you may need to prove what the state was.
- robots.txt, including rules addressed to AI-specific crawler names.
- Page-level signals such as robots meta tags and response headers.
- Machine-readable rights reservations. In the EU, rights holders can reserve text-and-data-mining rights in machine-readable form, and providers of general-purpose AI models are expected to respect those reservations and document their training content.
- Site terms that prohibit automated collection or reuse.
Store the opt-out state with each document, and keep the ability to remove documents later if a source withdraws permission. The wider framing is in ethical residential proxies for AI data collection.
Extraction: from HTML to text
Raw HTML is mostly not content. Navigation, footers, cookie banners, related-article widgets and ads can outnumber the actual text on a page.
- Extract the main content and discard boilerplate.
- Preserve structure that carries meaning: headings, lists, tables and code blocks.
- Handle rendered pages. Some content only exists after JavaScript runs; see when you need a web scraping API.
- Identify language per document, and per paragraph for mixed-language pages, so the mixture targets can actually be measured.
Keep the raw responses for a replay window. Extraction logic will improve, and re-extracting from stored responses is far cheaper than recrawling. The landing pattern is in moving web scraping API data into SQL.
Quality filtering
Most of what a crawl returns is not worth training on. Filtering usually runs in layers, cheapest first.
Heuristic filters remove obvious junk: very short documents, pages dominated by symbols or numbers, lines repeated many times, text without ordinary function words, and pages that are mostly lists of links.
Model-based quality filters score documents against examples of text you want more of. They are powerful and they encode a definition of quality, so check what they systematically exclude.
Generated-content filtering matters more every year, as covered in how to detect and filter AI-generated content in web datasets.
Safety filters apply whatever content policy the model requires.
Calibrate every filter per language. A threshold tuned on English will remove far too much from some languages and far too little from others. And measure what each stage removes, by language and domain, so a filter that is quietly deleting a whole region’s writing gets caught.
Deduplication at corpus scale
Duplicate text is everywhere on the web: syndicated articles, mirrored sites, templated pages and boilerplate repeated across a domain. Left in, it overweights whatever happens to be copied most.
- Exact duplicates are removed by hashing normalised text.
- Near-duplicates are found with MinHash and locality-sensitive hashing over shingled text, which scales to very large corpora.
- Repeated paragraphs within a domain, such as disclaimers and signatures, are removed at the paragraph level.
- Duplicates across time, where the same page appears in several crawl snapshots, are collapsed to one version.
Personal data
A web crawl collects personal data whether you want it or not: names, email addresses, phone numbers and sometimes identification numbers. Do not collect from behind logins, scrub common identifier patterns during processing, and document what the pipeline removes. The legal framing is in residential proxies and GDPR compliance.
Decontamination
If your evaluation benchmarks appear in the training data, your evaluations stop measuring anything. Benchmark questions and answers get copied across the web, so contamination happens by default.
Check the corpus for overlap with every evaluation set you plan to use, typically with long n-gram matching, and remove or flag the matches. Record which benchmarks were checked, so later results can be interpreted honestly.
Document and version everything
A training corpus is the product of hundreds of decisions, and nobody can use it responsibly without knowing them.
- Per-document provenance: source URL, fetch time, the vantage point it was observed from, opt-out state, and which filter versions it passed.
- A dataset card covering sources, time range, languages, filters and their versions, known gaps and known biases.
- A reproducible pipeline, so a corpus version can be rebuilt or amended when a source withdraws permission.
The case for recording where and when each observation was made is set out in the vantage-point standard.
The collection layer at scale
Large-scale collection needs distributed, geographically appropriate exits, both to reach regional sources and to keep per-host request rates reasonable. With the Shifter gateway, the market is set in the credentials against p.shifter.io:443, and omitting a session identifier rotates the exit per request, which suits a frontier fetching many independent pages:
customer-USERNAME-country-jp:PASSWORDWhy the proxy layer matters for training data specifically is covered in residential proxies for AI training data.
Metrics that keep the corpus honest
| Metric | What it tells you |
|---|---|
| Tokens by language and domain against target | Whether the mixture is being met |
| Removal rate per filter stage, by language | Whether a filter is over-deleting somewhere |
| Duplicate rate | How much of the crawl was repetition |
| Opt-out coverage | Share of documents with a recorded opt-out check |
| Contamination hits per benchmark | Whether evaluations can be trusted |
FAQ
Can we start from a public web crawl instead of crawling ourselves?
Public crawls are a strong starting point. They lag behind the live web and have coverage gaps, so most teams add targeted, fresher collection for the languages and domains that matter to them.
How aggressive should quality filtering be?
Aggressive enough to remove junk, measured carefully enough to see what else it removes. Audit removals by language and region before committing to thresholds.
Do we need to follow robots.txt for training data?
Yes, including AI-specific rules, and record the state at fetch time. Opt-outs are also becoming a legal expectation in some jurisdictions.
How is this different from grounding data?
Training data shapes the model’s weights. Grounding data is fetched at answer time and cited. The second is covered in grounding LLM agents with live web data.
The bottom line
A large-scale training corpus is built in the stages after the crawl: a mixture target that decides what to collect, a frontier that stays polite, opt-outs checked and recorded at fetch time, clean extraction, layered filtering calibrated per language, deduplication at every level, decontamination against evaluations, and documentation that lets anyone reconstruct the decisions.
Collect from the regions the mixture needs, keep raw responses for replay, and measure what every stage removes. The product view is on the residential proxies for AI and ML page, with rates on the pricing page.