Every AI team reaches the same fork early: license a dataset someone else has built, or collect the data yourself. Marketplaces make buying look like a one-click decision, and for some projects it is. For others, licensed data turns out to be stale, thin in the places that matter, or tied to terms that block the product you meant to ship. This guide is for teams building models from web data who want to make that call on purpose rather than by default.
We cover what the market for training data looks like, what a licence really gives you, the costs that do not show on the price tag, and a framework for deciding, with the hybrid setup most mature teams end up running.
What the training data market looks like
“Marketplace” covers several quite different ways to get data:
- General data exchanges. Cloud providers and data platforms run catalogues where vendors list datasets and buyers subscribe, with delivery straight into the buyer’s storage or warehouse. Most listings were built for analytics; training use has to be allowed explicitly in the licence.
- Specialist AI data vendors. Companies that sell curated text, image, audio and speech collections, often with annotation, and increasingly custom collection on request.
- Open dataset hubs. Repositories where datasets are free to download but each carries its own licence, and the licence field is not always filled in or correct.
- Direct licensing deals. Large platforms and publishers now license their content to AI developers directly. Reddit’s 2024 agreement with Google and a series of news publisher deals with AI labs made this a visible part of the market, but it is a route for large buyers, not a self-serve catalogue.
The common thread is that you are buying rights as much as bytes. Two datasets with the same contents can be worth very different amounts depending on what the licence lets you do.
What a licence actually gives you
Before comparing prices, read the licence for four things:
- Permitted use. Training, fine-tuning, evaluation and commercial deployment are separate permissions. “Research use” usually excludes a commercial model.
- Scope and exclusivity. Is the licence per model, per product or company-wide? Can the vendor sell the same data to your competitors? Almost always, yes.
- Term and what happens after it. Many licences are annual. Check whether a model trained during the term may still be used after it ends, and whether you must delete the raw data.
- Warranties. Does the vendor confirm it had the right to collect and sell the data, and will it stand behind that claim? A licence without that warranty moves the legal risk back to you.
Open hubs deserve the same scrutiny. The Data Provenance Initiative’s audit of widely used text datasets found licence information missing or wrong in a large share of cases, which is why we recommend keeping your own record of where every document came from, as described in our guide to provenance in the AI training pipeline.
The costs that are not on the price tag
Freshness. A purchased dataset is a snapshot. If your model has to know current prices, products, regulations or language, the snapshot starts ageing on the day you buy it, and refresh subscriptions are priced accordingly.
Coverage gaps. Catalogues are strongest in English and in the largest markets. Teams that need a specific country, a regional language or a niche domain often find the data thin, or sampled in ways the vendor does not document.
Shared data. If you can buy it, so can everyone else. Licensed data rarely gives a model an edge on its own.
Integration. Every vendor has its own schema, deduplication rules and quality level. Cleaning and joining sources is real engineering work, whichever way the data arrived.
Compliance paperwork. The EU AI Act summary template for general-purpose models asks providers to list licensed private datasets alongside public datasets and scraped sources. Buying data does not remove documentation work; it changes what you have to document.
The costs of collecting it yourself
Collection has its own bill:
- Engineering. Crawlers, parsers, scheduling, storage and monitoring, plus maintenance as target sites change.
- Access. Sites limit repeated requests from a single IP and often show different content by country. Gathering public data at scale, from the right locations, needs a network built for it, such as residential proxies with country and city targeting.
- Policy. You need a clear rule for robots.txt, text and data mining reservations and site terms, recorded at the moment of collection. Our overview of robots.txt and AI opt-out signals covers the signals to respect.
- Quality work. Deduplication, language identification, boilerplate removal and filtering take as long as the crawl itself.
In return you get data that is current, shaped to your task, documented the way you need, and not sitting in a competitor’s training mix.
A framework: when to buy, when to collect
| Situation | Better choice | Why |
|---|---|---|
| One-off benchmark or evaluation set | Buy or use open data | Bounded, and freshness matters less |
| Data must be updated weekly or daily | Collect | Refresh licences get expensive and lag |
| Specific countries or regional languages | Collect, or buy and top up | Catalogues are thin outside large markets |
| Data that only exists behind a platform’s walls | License directly | Collection would break terms or be impossible |
| Proprietary edge is the point | Collect | Licensed data is available to everyone |
| Small team, first prototype | Buy or use open data | Ship first, invest in collection later |
| Audit-heavy customers or regulators | Either, with full records | What matters is the record, not the route |
A useful test: if you would need to buy the same dataset again in six months, you are paying for a pipeline without owning one.
The hybrid most teams end up with
Mature teams rarely pick one side. A common pattern:
- Start from open and licensed data to get a working baseline quickly.
- Measure where the model fails: missing countries, stale facts, weak domains.
- Collect only for those gaps, from public sources, on a schedule, with provenance recorded at fetch time.
- Keep one catalogue that lists every source, its licence or collection policy, and its refresh date, so the public summary and customer questionnaires take hours rather than weeks.
If you are building that collection layer, our guide to building large-scale training datasets from the open web walks through the pipeline from crawl to clean records.
The bottom line
Buy training data when the need is bounded, common and not time-sensitive, and read the licence as carefully as the sample. Collect it yourself when freshness, regional coverage or a proprietary edge matters, and record provenance from the first request. Most teams do both, and the ones that do it well treat every source, bought or collected, as something they can explain.