For most of the web’s history, public data was simply there to be collected, and the only question was whether you could. That is changing. Platforms, publishers and data owners increasingly put their data behind licences, paid APIs and formal data access agreements, and they write terms aimed squarely at automated collection and AI training.
For organisations that collect web data, this is not only a restriction. A data access agreement can give you cleaner data, clear rights and less legal risk than collecting it yourself. For organisations that publish data, it is a way to decide who uses it and on what terms. This guide covers what these agreements contain, when each side should use one, and how they sit alongside ordinary collection of public pages.
This is general information, not legal advice. Agreements and the law around them vary, and counsel should review any agreement before you sign it.
Why data access agreements are spreading
Three changes pushed data owners towards formal agreements:
- AI training raised the value of data. Text, images and structured data became training material, and owners started asking to be paid for it or to set the terms of its use.
- Major platforms moved access behind paid APIs. In 2023 several large social platforms introduced or sharply raised charges for API access that had previously been free or cheap, turning access into a commercial relationship.
- Opt-out signals gained legal weight. In the EU, rightholders can reserve their content from commercial text and data mining in a machine-readable way, and AI providers must respect those reservations. Our guide to robots.txt, AI opt-outs and reservation signals covers how that works.
The result is a web with more explicit terms: some data stays freely collectable, some is available only under licence, and the boundary is increasingly written down.
What a data access agreement covers
Agreements differ, but most deal with the same questions:
| Clause | What it settles |
|---|---|
| Scope | Which data, which fields, which history, how fresh |
| Delivery | API, bulk files or a feed, and in what format |
| Permitted use | Internal analysis, products, resale, AI training, each allowed or not |
| Volume and rate | How much, how often, and what happens above the limit |
| Attribution and display | Whether and how the source must be credited |
| Personal data | Who is the controller, what may be kept, how deletions flow through |
| Term and termination | How long it lasts, and what must be deleted at the end |
| Fees | Fixed, per call, per record or revenue share |
| Audit and reporting | What the licensee must show about its use |
The most important clause for most buyers is permitted use. An agreement that allows internal analytics may forbid resale or model training, and using the data outside its scope can be a breach even when the data itself is public.
For data collectors: when an agreement beats collecting
Collecting public pages remains legitimate and is often the right choice. But an agreement is worth considering when:
- You need data that is not public, such as history, identifiers or fields that never appear on a page.
- You need clear rights to resell or train on the data. An agreement can grant rights that collection never could; our guide to who owns scraped data explains why collection alone does not give you ownership.
- The source blocks or changes often, and maintaining a collector costs more than a licence.
- Your customers or compliance team need provenance that a signed agreement documents better than a crawl log.
And collection remains the better fit when you need the public view as users see it, across many sources, in many markets, where no agreement exists or the source has no interest in offering one. Seeing each market as a local visitor does is what residential proxies are for.
Best practices if you sign one:
- Map permitted use to every use you plan, including future products and AI work, before signing.
- Keep licensed and collected data separate, with provenance on every record, so you always know which terms apply.
- Plan for termination: know what you must delete and design storage so you can.
- Watch for exclusivity and most-favoured terms that limit what else you may do.
For data owners: regaining control of your data
If you publish data that others collect, an agreement is one tool among several for deciding how it is used:
- Decide what is genuinely public. Content you want indexed and seen is hard to also keep from collectors; content you want to license may belong behind an API or a login.
- State your terms clearly and machine-readably. Terms of use, robots.txt and, in the EU, machine-readable reservations for text and data mining all signal your position. Clear signals make agreements easier to negotiate and breaches easier to show.
- Offer a legitimate route. A documented API or licence gives serious users a way to pay and comply instead of collecting around you, and gives you a record of who uses your data.
- Price for the use, not just the volume. Analytics, resale and AI training have very different value.
- Match enforcement to reality. Technical measures, rate limits and contract claims each have limits; courts have treated claims about how data was accessed differently from claims about who owns public data.
The legal backdrop, briefly
Agreements sit on top of a legal framework that is still settling. In the US, courts have held that a platform’s terms did not stop logged-off collection of public data, while claims based on how data was accessed, such as fake accounts or server burden, have survived. In the EU, contract terms can restrict the use of databases that are not otherwise protected, the sui generis database right protects substantial investment in databases, and data protection law applies to personal data however it was obtained.
Before relying on either collection or an agreement, legal teams ask the same questions about scope, access, rights and personal data; our checklist of what your legal team will ask before approving a scraping project covers them.
FAQ
What is a data access agreement?
A contract under which a data owner gives another party access to its data, usually through an API, files or a feed, on stated terms: what data, how it may be used, how much, for how long and at what price.
Is it better to license data or scrape it?
It depends on what you need. Licensing suits non-public data, clear resale or AI training rights, and sources that change often. Collecting public pages suits the as-seen view across many sources and markets where no agreement is offered.
Can a data access agreement restrict data that is public?
It can restrict how the licensee uses the data it receives under the agreement, even data that is also public. Whether a site’s general terms restrict collection of its public pages by others depends on the jurisdiction and whether those terms bind the collector.
How can a website control how its data is used?
By deciding what to make public, stating terms and opt-outs clearly and machine-readably, offering a licensed route for legitimate users, and enforcing against collection that breaches its terms or harms its service.
The bottom line
Data access agreements are becoming a normal part of how web data changes hands. For collectors, they offer rights and data that collection cannot, at a price; for owners, they offer a way to decide who uses their data and how. Use agreements where they give you something collection cannot, keep licensed and collected data apart, and read the permitted-use clause before anything else.