Provenance in the AI Training Pipeline: Proving Where Your Dataset Came From

Ask an AI team where a given training document came from and the answer is often a dataset name, not a source. Regulators, customers and courts now want the second answer: which page, fetched when, under which terms, and what was done to it before training. For teams that build models from web data, provenance has moved from good practice to something you have to be able to show.

The hard part is timing. Most of what a provenance record needs exists only at the moment of collection. This guide covers why the question became pressing, the record worth keeping for every document, and how to carry it through extraction, filtering and deduplication without losing the thread.

Why provenance became a requirement

Three developments turned provenance from a research topic into a compliance task.

The EU AI Act asks for a public summary. Article 53(1)(d) requires providers of general-purpose AI models to publish “a sufficiently detailed summary about the content used for training”, using a template from the AI Office. The obligations apply from 2 August 2025, and models already on the market before that date have until 2 August 2027. The Commission presented the template in July 2025 and states that using it is mandatory.

The template asks for specifics. For scraped data it asks for the names of the crawlers used, the period of collection, and a list of the most relevant domain names: the top 10% of all domains by the size of the content scraped, or for small and medium-sized companies the top 5% or 1,000 domains, whichever is lower. It also asks providers to describe how they respected text and data mining opt-outs “before and during data collection”. Article 53(1)(c) separately requires a copyright policy that identifies and complies with those reservations.

The commons is shrinking, and the records were poor. The Data Provenance Initiative audited more than 1,800 text datasets in 2023 and found licence omission rates above 70% and error rates above 50% on widely used hosting sites. A follow-up audit of 14,000 web domains found that in a single year, about 5% of all tokens in C4, and 28% of its most actively maintained sources, had become fully restricted by robots.txt. A dataset that was acceptable when crawled may not be acceptable to crawl today, and you can only show the difference if you recorded the state at the time.

What to record for every document

A provenance record is written once, by the crawler, at fetch time. Everything downstream adds to it rather than replacing it.

Field	Why it matters
Source URL	The basis of the domain list and of any takedown request
Fetch time (UTC)	Proves the collection period and the opt-out state on that date
Response status	Separates real content from block pages and errors
Hash of the raw body	Ties every later copy to exactly what was received
Vantage point	The same URL can serve different content by country and network
Robots.txt decision	Shows the crawler honoured the rules in force at fetch time
TDM reservation seen	The opt-out the EU copyright rules and the template care about
Stated licence or terms	What the page said about reuse, if anything
Crawler name and version	The template asks for crawler names; versions explain behaviour changes
Parent records	Links a cleaned or deduplicated copy back to its source

Two fields are often missed. The first is the vantage point: a page observed from one country or network can differ from the same URL seen elsewhere, which is the argument of our piece on recording where web data was observed. The second is the raw body itself. A hash proves which bytes you received, but keeping the response, for example by archiving pages in WARC files, lets you show it.

A provenance record in code

The example below writes one record per fetched document, keeps only documents that were allowed and not reserved at fetch time, and links each extracted copy back to its source by hash. It also produces the domain ranking the summary template asks for. It runs on Python 3.9 and later with the standard library only.

from __future__ import annotations

import hashlib
import json
from collections import Counter
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from urllib.parse import urlsplit


def sha256(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


@dataclass
class Provenance:
    """One record per fetched document, written at collection time."""
    url: str
    fetched_at: str            # ISO 8601, UTC
    status: int
    content_sha256: str        # hash of the raw response body
    vantage: str               # where it was observed from, e.g. "residential:de"
    robots_allowed: bool       # robots.txt decision for our crawler, at fetch time
    tdm_reserved: bool         # machine-readable TDM opt-out seen, at fetch time
    license: str | None        # licence or terms as stated on the page, if any
    collector: str             # crawler name and version
    parents: list[str] = field(default_factory=list)  # hashes this record derives from
    step: str = "fetch"        # fetch, extract, filter, dedup ...


def fetched(url, body, status, vantage, robots_allowed, tdm_reserved, license=None,
            collector="corpus-crawler/1.4"):
    return Provenance(
        url=url,
        fetched_at=datetime.now(timezone.utc).isoformat(timespec="seconds"),
        status=status,
        content_sha256=sha256(body),
        vantage=vantage,
        robots_allowed=robots_allowed,
        tdm_reserved=tdm_reserved,
        license=license,
        collector=collector,
    )


def derived(parent: Provenance, new_body: bytes, step: str) -> Provenance:
    """A transformed copy keeps the source fields and points back at its parent."""
    rec = Provenance(**{**asdict(parent), "parents": [parent.content_sha256], "step": step})
    rec.content_sha256 = sha256(new_body)
    return rec


def eligible(rec: Provenance) -> bool:
    """Only documents that were allowed and not reserved at fetch time go further."""
    return rec.status == 200 and rec.robots_allowed and not rec.tdm_reserved


def domain_summary(records, top_share=0.1):
    """Domains by document count, and the top share of them, for a training-data summary."""
    counts = Counter(urlsplit(r.url).hostname for r in records)
    ranked = [d for d, _ in counts.most_common()]
    keep = max(1, round(len(ranked) * top_share))
    return counts, ranked[:keep]


if __name__ == "__main__":
    raw = [
        fetched("https://news.example/a", b"<html>a</html>", 200, "residential:de", True, False),
        fetched("https://news.example/b", b"<html>b</html>", 200, "residential:de", True, False),
        fetched("https://shop.example/p/1", b"<html>p1</html>", 200, "residential:fr", True, True),
        fetched("https://blog.example/x", b"<html>x</html>", 403, "residential:us", True, False),
        fetched("https://forum.example/t/9", b"<html>t9</html>", 200, "residential:us", False, False),
    ]
    kept = [r for r in raw if eligible(r)]
    texts = [derived(r, r.url.encode() + b" text", "extract") for r in kept]

    with open("manifest.jsonl", "w") as f:
        for rec in raw + texts:
            f.write(json.dumps(asdict(rec)) + "\n")

    counts, top = domain_summary(raw)
    print(f"fetched {len(raw)}, eligible {len(kept)}, extracted {len(texts)}")
    print("domains:", dict(counts))
    print("top domains:", top)
    child = texts[0]
    print("lineage ok:", child.parents == [kept[0].content_sha256], child.step, child.url)

Output:

fetched 5, eligible 2, extracted 2
domains: {'news.example': 2, 'shop.example': 1, 'blog.example': 1, 'forum.example': 1}
top domains: ['news.example']
lineage ok: True extract https://news.example/a

The reserved page, the blocked page and the page disallowed by robots.txt are all still in the manifest. That matters: the record of what you chose not to use is part of the proof. The template ranks domains by the size of content scraped, so in production, count bytes or tokens rather than documents.

Carrying provenance through the pipeline

A record written at fetch time is only useful if every later step keeps the link.

• Extraction. The text you train on is not the HTML you fetched. Give the extracted text its own hash and point it at the raw record, as derived() does above.
• Filtering. Log why a document was dropped as well as that it was dropped. “Below quality threshold” and “TDM reservation” are different answers to a regulator.
• Deduplication. When near-duplicates are merged, keep the list of every source the surviving copy stands for. Otherwise a takedown request for one URL cannot find the document it ended up in.
• Mixing and sampling. Record which manifest version each training run used, so the summary describes the data a model actually saw.
• Removal. Keep a drop-list keyed by URL and hash. When a site later reserves its rights or a rights holder asks for removal, the same keys find the document in every derived dataset.

Opt-out state deserves particular care. A robots.txt file or a text and data mining reservation can change after you crawl, in either direction. Record the decision you made at fetch time, and re-check before each new collection rather than assuming last year’s answer still holds.

Standards worth borrowing

You do not need to invent a schema from scratch.

• Datasheets for Datasets (Gebru et al., Communications of the ACM, 2021) proposes that every dataset document its motivation, composition, collection process and recommended uses. It is the right shape for the dataset-level summary.
• The Data Provenance Standards from the Data & Trusted AI Alliance define 22 metadata fields in three groups (source, provenance and use), including the date range of data generation, the method, the licence to use and where consent is documented. Version 1.0.0 was released in July 2024.
• C2PA Content Credentials attach a signed provenance manifest to an individual media asset. They are most useful for images, audio and video, where the asset itself can carry its record.

Where provenance usually breaks

• It is added after the fact. A crawl without per-document records cannot be audited later, however good the dataset card is.
• The licence is copied from the dataset card. The audit above found this is where errors concentrate. Record what the source said, and treat aggregated labels as claims to check.
• Block pages are counted as content. A 200 response can still be a challenge page. Status codes and content checks belong in the record.
• The vantage point is unknown. Collecting each market from a local residential connection only helps provenance if the record says which country and network each fetch used.

The bottom line

Provenance is cheap to write at collection time and close to impossible to recover afterwards. Write one record per document at fetch time, carry the hash through every transformation, keep what you excluded as well as what you kept, and the summary the AI Act asks for becomes a query instead of an investigation.

Sources and references

• Regulation (EU) 2024/1689 (AI Act), Articles 53, 111 and 113: eur-lex.europa.eu
• European Commission, explanatory notice and template for the public summary of training content (July 2025): digital-strategy.ec.europa.eu
• Directive (EU) 2019/790, Article 4(3): eur-lex.europa.eu
• Longpre et al., The Data Provenance Initiative (2023): arxiv.org/abs/2310.16787
• Longpre et al., Consent in Crisis: The Rapid Decline of the AI Data Commons (2024): arxiv.org/abs/2407.14933
• Gebru et al., Datasheets for Datasets: doi.org/10.1145/3458723
• Data & Trusted AI Alliance, Data Provenance Standards v1.0.0: dtaalliance.org
• C2PA Specification: spec.c2pa.org