Most of the open web is written in a handful of languages, and a crawler that runs from one country sees an even narrower slice: the English or default version of sites that would show local text to a local visitor. If you train or fine-tune models for several markets, collecting genuinely multilingual and regional data is its own problem, and it is where AI and ML data collection gets hard.
This guide covers how to plan coverage by language, where to find local sources, how to fetch the version local readers see, and how to identify, filter and balance the text once you have it. For the general pipeline from crawl to corpus, see our guide to building large-scale training datasets from the open web.
Plan coverage by language and region
Start with a table, not a crawler. For each target language, write down:
- Regions that matter. Spanish from Spain, Mexico and Argentina differ in vocabulary, spelling and topics. Portuguese from Brazil and Portugal differ more. Decide which varieties you need.
- Target volume per language and variety, in tokens or documents.
- Domains you need covered: news, forums, government, commerce, reference, technical documentation.
- Scripts. Some languages are written in more than one script, such as Serbian in Cyrillic and Latin. Decide whether you keep both.
This table becomes your test. At the end of each collection run, compare what you got against it, language by language.
Find local sources
General crawls under-sample small languages, so build seed lists per language:
- Country-code domains such as
.pl,.vnor.co.keare a strong starting signal for regional content. - Language versions of global sites.
hreflangtags in page headers and sitemaps list every language version of a page. Our guide to using hreflang to find every language version of a page shows how to extract them. - Local directories, news aggregators and government portals, which link out to many sites in the same language.
- Search results in the local language, run from the region, which surface sites a global crawl rarely reaches.
Record the source and the opt-out status for every seed. Respect robots.txt and AI opt-outs at fetch time, and keep that record with the data.
Fetch the version local readers see
Many sites decide what to serve from the visitor’s location: language, currency, local news, a regional edition or a consent wall. A crawler in one data center gets the default version for every country, often English, and records it as if it were local content.
To collect the local version, each request needs to look like a local reader:
- An IP in the target country, ideally a home connection. Requests through residential proxies arrive from residential IPs in the chosen country or city, so sites route and localize them the way they do for real visitors.
- A matching
Accept-Languageheader, for examplepl-PL,pl;q=0.9for Polish readers in Poland. - Consistent signals. Time zone and locale in a headless browser should match the IP. Mismatches are one reason sites fall back to a default version; our post on matching proxy geo, time zone and locale goes through the checks.
Check a sample from each region by hand early in the run. If the “Vietnamese” bucket is full of English pages, the fetch layer is wrong, and no filter later will fix it.
Identify the language of every document
Never trust the lang attribute or the domain alone. Run a language identification model on the extracted text:
- fastText’s lid.176 model covers 176 languages and is fast enough to run on every document.
- Broader models such as GlotLID cover far more languages and are better for low-resource ones.
Run identification per paragraph as well as per document. Many pages mix languages: a local article with an English navigation menu, or a forum thread where users switch. Keep the paragraphs in the language you want, and record a confidence score so you can tighten the threshold later. Short texts are where identification fails most, so set a minimum length before you trust a label.
Filter per language, not globally
Quality filters tuned on English damage other languages. Heuristics such as stop-word counts, average word length or punctuation ratios behave differently in Thai, which has no spaces between words, or in Finnish, which has very long words. Set thresholds per language, from a sample a native speaker has reviewed.
Two problems grow in multilingual data:
- Machine-translated content. Many sites publish automatic translations of English pages. They add volume but teach the model translation errors. Flag pages that are near-translations of known English pages, and treat whole sites that translate everything automatically with suspicion.
- Encoding errors. Wrongly decoded text produces mojibake that looks like a real but strange language. Normalize to Unicode, repair or drop broken encodings, and normalize Unicode forms so the same word is not stored two ways.
Deduplicate within each language and across languages, because the same boilerplate, legal notices and templates appear in every version of a site.
Balance the mix
Raw web data is dominated by a few languages. Training on raw shares means small languages are barely learned. Training on equal shares means small languages are repeated many times and the model memorizes them.
The common middle path is temperature sampling: each language is sampled in proportion to its size raised to a power below one. The mT5 and XLM-R papers used an exponent of 0.3. Whatever you choose, write it down with the per-language token counts before and after sampling, so the mix can be reproduced and explained. Our guide to provenance in the AI training pipeline covers what to record.
The bottom line
Multilingual data is not English data with a translation step. Plan coverage by language and variety, find local sources, fetch each region the way local readers see it, identify language per paragraph, filter with per-language thresholds and balance the mix on purpose. The collection layer decides whether your “Polish” data is really Polish, so get the fetch right first.