Wikipedia is one of the most heavily used sources for AI training and answer engines, and its operator has been saying with growing firmness how it wants that to happen. For teams that build models and AI products from web data, this matters beyond one website: it is the clearest example so far of a major open source of knowledge setting out what it expects from AI companies, and then enforcing it.
The story has moved on since the first headlines. This piece covers what the Wikimedia Foundation actually asked for, how it is enforcing it, and the practical routes to the data today.
What Wikipedia actually asked for
On 10 November 2025, the Foundation asked AI developers for two things: “attribution and financial support”. The second meant that most AI developers “should properly access Wikipedia’s content through the Wikimedia Enterprise platform”, which it described as a “paid-for opt-in product”. That post did not use the word “scraping”; the headline phrase came from how it was reported.
The language hardened during 2026. In July the Foundation wrote that bad-faith actors “continue to tax our public API infrastructure to crawl and scrape”, and that for large-scale users Enterprise is “a must-do rather than a nice-to-have”. It also said it is not switching off public API access, “now or ever”. And in an October 2026 court filing post it wrote that scraping of its projects “must be done sustainably and in line with our public API rate limits or through our Enterprise API offerings”.
Why now
The Foundation’s reasons come down to two numbers moving in opposite directions.
People are visiting less. In October 2025 it reported human pageviews down roughly 8% against the same months of 2024, after improving its bot detection when it found unusual traffic, mostly from Brazil, that turned out to be bots built to evade detection. It asks readers to interpret the figure with care, and it updated its detection again in October 2026, so later numbers may move.
Machines cost more. In April 2025 it reported that bots made up about 35% of pageviews but at least 65% of its most resource-consuming traffic, because they fetch unpopular pages that are not cached. Bandwidth for multimedia downloads had grown 50% since January 2024. As the post put it: “Our content is free, our infrastructure is not”.
Fewer visitors also means fewer potential editors and donors, which is why the Foundation treats attribution as more than a courtesy.
How it is being enforced
This is no longer only a request. By April 2026 the Foundation said it was blocking or throttling about 30% of automated requests from crawlers that break its policies. Global API rate limits finished rolling out in June 2026.
| Rule | What it says |
|---|---|
| API, no identification | 10 requests a minute |
| API, identifying User-Agent, no login | 200 requests a minute |
| Crawling (robot policy) | Honour robots.txt, average below 20 requests a second, under 10 concurrent |
| Browser user agents on bots | Bot-like behaviour “will be assumed malicious” |
| Dumps site | 3 connections per IP |
| Higher volume | External users are pointed to Enterprise |
The routes to the data
You have more legitimate options than the headline suggests.
| Route | Good for | Limits |
|---|---|---|
| Public APIs | Small, identified lookups | The rate limits above |
| Enterprise free account | Prototypes and moderate use | 50,000 On-demand requests and 30 Snapshots a month, monthly updates |
| Enterprise paid | Production, freshness, Realtime | Prices on request |
| Kaggle dataset | Experiments with structured JSON | Beta, English and French |
| Dumps and content exports | Full offline copies | XML dumps end in January 2027 |
| Wikidata | Structured facts | Released under CC0, so no attribution needed |
Commercial uptake is real but still small. In January 2026 Wikimedia Enterprise named Amazon, Meta, Microsoft, Mistral AI and Perplexity as new partners, joining Google and others. For its 2024-25 fiscal year it reported $8.3 million in revenue from 13 commercial customers on the paid tier.
Access is not the same as a licence
One detail is easy to miss. The Foundation says Enterprise’s commercial strategy “isn’t licensing-based. It’s access-based”. Paying buys reliable delivery, not different rights. The text remains under CC BY-SA 4.0, whichever route you use: reuse needs attribution, such as a link to the page, and modified text has to carry the same licence. Facts themselves can be reused freely.
Whether training on the text triggers those terms is unsettled. In an amicus brief in the New York Times case against OpenAI, published 30 October 2026, the Foundation argued that training a new and different model may not require meeting licence terms, while answer-engine use that retrieves and repeats content “should not be considered fair”. That is a party’s argument, not a ruling, and it may not reflect how a court decides. Our guide to who owns scraped data covers the wider questions, and counsel should review your specific use.
The practical point stands either way: retrieval products have the strongest reason to carry attribution. The Foundation has said a dedicated Attribution API is among its next steps, so it is worth designing for this now.
What AI and data teams should do
- Find out how you use it today. Check your crawler logs and vendor contracts. If Wikipedia reaches you through a third-party dataset, ask how that data was collected.
- Pick the route by need. A one-off corpus may only need an export or a Snapshot. A live answer engine probably needs Enterprise.
- Identify your client. Use a descriptive User-Agent with contact details, stay inside the published limits, and honour robots.txt.
- Plan for the dumps change. If a pipeline depends on the XML dumps, move it before January 2027.
- Keep source records. Store the page URL and revision for every document, so attribution and removal stay possible. Our guide to provenance in the AI training pipeline shows how.
- Budget for access as a line item. Wikipedia is part of a wider shift, alongside crawler tolls and licensing standards.
A note on residential proxies
The Foundation names residential proxies among the ways some crawler operators get around its limits. We sell residential proxies, so to be plain: do not use them for this. Wikipedia offers bulk data, free tiers and a documented path to higher volume, and it has asked for bulk traffic to take that path. There is no good reason to disguise a crawler here, and its policies say such clients may be blocked.
Residential proxies are for a different job, such as seeing how public pages appear to local visitors in different markets. The right tool for Wikipedia’s data is the one Wikipedia provides.
The bottom line
Wikipedia’s message has grown from a request into policy backed by rate limits and blocking, with an open door for anyone who takes the official route. For most teams the answer is simple: use the exports, free tier or Enterprise, identify your client, keep records for attribution, and watch how the courts treat the licence question. It is a useful template for how other open sources may behave next.
Sources and references
- Wikimedia Foundation, In the AI era, Wikipedia has never been more valuable (10 November 2025): wikimediafoundation.org
- Wikimedia Foundation, on Wikimedia Enterprise and protecting Wikipedia (16 July 2026): wikimediafoundation.org
- Wikimedia Foundation, amicus brief in New York Times v. OpenAI (30 October 2026): wikimediafoundation.org
- Diff, New user trends on Wikipedia (17 October 2025): diff.wikimedia.org
- Diff, How crawlers impact the operations of the Wikimedia projects (1 April 2025): diff.wikimedia.org
- Diff, Quo vadis, crawlers? (26 March 2026): diff.wikimedia.org
- Diff, Wikimedia Enterprise financial report, fiscal year 2024-2025: diff.wikimedia.org
- Wikimedia Enterprise, partners (15 January 2026) and pricing: enterprise.wikimedia.com, enterprise.wikimedia.com/pricing
- Wikimedia Enterprise, Kaggle dataset: enterprise.wikimedia.com
- Wikimedia APIs, rate limits: mediawiki.org
- Wikimedia robot policy: wikitech.wikimedia.org
- Wikimedia User-Agent policy and Terms of Use: foundation.wikimedia.org, foundation.wikimedia.org
- Wikimedia dumps: dumps.wikimedia.org