For most of the web’s history, a crawler paid for bandwidth and compute, and the content itself cost nothing. That assumption is starting to break. Sites can now answer an AI crawler with a price, and teams that collect web data for AI and machine learning need to work out what that does to their budgets.
The change is real but early. This guide covers how crawler tolls work, what they add to the cost of a crawl, and how to plan for a web where some pages are free, some are priced and some are closed. It is general information, not legal advice.
How a crawler toll works
The idea uses an old placeholder. HTTP has had a 402 Payment Required status for decades, and RFC 9110 still says it “is reserved for future use”. Cloudflare gave it a use in July 2025 with pay per crawl, which its documentation still describes as a closed beta.
The exchange is simple:
- A verified AI crawler requests a page without payment terms, and receives a 402 with a
crawler-priceheader. - It retries with a
crawler-exact-priceheader to accept that price, or it sends acrawler-max-priceheader up front to say what it will pay. - It gets the page with a 200 and is billed. Cloudflare acts as merchant of record.
For each crawler, a site owner can charge, allow or block. The price applies to the whole domain by default and can be overridden, with a minimum of $0.001 per crawl.
Two details matter more than the price. First, the crawler has to prove who it is: Cloudflare requires Web Bot Auth, which signs each request with the crawler’s own key using HTTP Message Signatures (RFC 9421). An IETF working group is standardising the method, and the protocol is still a draft. Second, for a crawler with no billing relationship, Cloudflare describes a charge as “the functional equivalent of a network level block”. In practice, a toll is a block with a price on it.
It is more than one toll booth
Cloudflare is the most visible player, but the market is broader and moving quickly.
- Licensing terms in robots.txt. The RSL standard, launched in September 2025 with backers including Reddit, Yahoo, People Inc., Ziff Davis, O’Reilly Media and Medium, adds machine-readable licensing and royalty terms to robots.txt, with models including pay-per-crawl and pay-per-inference. Version 1.0 was published in December 2025.
- Paywalls at the network edge. TollBit’s bot paywall integrates with Fastly and Akamai, and Akamai also works with Skyfire on payment tokens for agents.
- Marketplaces. Microsoft described a publisher content marketplace in February 2026 in which “publishers will be paid on delivered value”.
- Payment by use. In July 2026 Cloudflare said it is evolving pay per crawl into Pay Per Use, so publishers are paid when content creates value and not just when it is fetched. Per-request pricing may not be the final form.
Defaults are shifting too. Since September 2026, Cloudflare offers new domains one of two presets depending on whether the site earns money from advertising. For ad-funded sites, that means search allowed, AI training disallowed and agents blocked on pages with ads. Our guide to robots.txt, AI opt-outs and reservation signals covers how those signals fit together.
Why publishers are doing it
The motive is the broken exchange of crawling for traffic. Cloudflare’s July 2025 figures show about 38,066 crawls per referral for Anthropic, 1,091 for OpenAI and 195 for Perplexity, though it cautions that these calculations may overstate the ratios. TollBit reported 179 AI bot visits per human referral on European sites in the first half of 2026, against 53 in North America. Both companies sell these tolls, so read their numbers with that in mind, but the direction is not in much doubt.
What a toll does to your costs
Without tolls, the main variable cost of a crawl is bandwidth and infrastructure. A toll adds a charge per page, whatever the page weighs. The table below assumes pages of 1 MB and residential bandwidth at $0.75 per GB, our Full Geo rate. The $0.001 is Cloudflare’s minimum; the $0.01 is only an example, because no independent average price has been published.
| Crawl | Bandwidth | Toll at $0.001 | Toll at $0.01 |
|---|---|---|---|
| 1,000 pages | $0.75 | $1 | $10 |
| 1 million pages | $750 | $1,000 | $10,000 |
| 10,000 pages, daily for 30 days | $225 | $300 | $3,000 |
Three things follow.
The toll can overtake everything else. Even at the minimum, it is larger than the bandwidth for a 1 MB page. At $0.01 it is more than ten times larger, and any saving on bandwidth barely moves the total.
Recrawling gets expensive. A toll is charged per fetch, so freshness has a direct price. Monitoring jobs that refetch unchanged pages are the most exposed. Sitemaps, conditional requests and change detection become budget tools, not only efficiency work.
Failed work costs more. If a parser breaks after you paid for the pages, you need raw copies or you pay again. Store what you buy. Our piece on cost per clean record shows how to build this into your unit economics.
How to plan for it
- Audit your sources. Log which domains in your crawl return 402s, carry RSL terms or have tightened their robots.txt. TollBit reported that 30% of AI bot scrapes in Q4 2025 did not abide by explicit robots.txt permissions, a figure that shows how contested the signals are. Decide your own policy instead of copying the average.
- Cap spend per domain. Treat price as a crawl setting. A
crawler-max-priceheader or an equivalent budget stops one expensive site from consuming the whole run. - Rank sources by value. Ask what each domain adds to the dataset. Paying for the 20 sources that matter is usually better than crawling 2,000 thinly.
- Compare a toll with a licence. At volume, a negotiated agreement can be cheaper than per-request fees and can grant rights a toll does not. Our guide to data access agreements covers what to look for. Ask what a payment actually buys: a toll pays for access, and may say nothing about training rights.
- Build identity in. Signed requests from a known crawler are becoming the entry ticket. If you run a crawler that wants to pay, plan for key management and a published identity.
Where proxies fit, and where they do not
Tolls target crawlers that identify themselves, and paying traffic has to be signed. A proxy does nothing for that. A proxy changes where a request comes from, not what you are allowed to do, so if a site has set a price or reserved its content, a different IP does not change those terms.
Residential proxies remain useful for a different job: seeing public pages as local users see them, across markets, for work such as price monitoring, ad verification and QA. That work does not depend on avoiding anyone’s price, and it should respect the same signals.
The legal backdrop points the same way. The US Copyright Office’s 2025 pre-publication report said commercial use of vast troves of copyrighted works to produce competing content goes beyond established fair use boundaries, “especially where this is accomplished through illegal access”. How data is obtained matters as well as what is done with it.
The bottom line
Crawler tolls are still early: the leading product is in closed beta, prices are unsettled and the model may shift from paying per fetch to paying per use. But the direction is clear enough to plan for. Log which sources set terms, cap spend per domain, keep raw copies of what you pay for, and compare tolls with licences at volume. Teams that treat content as a line item now will not be surprised when it becomes one.
Sources and references
- Cloudflare, Introducing pay per crawl (1 July 2025): blog.cloudflare.com
- Cloudflare docs, What is Pay Per Crawl: developers.cloudflare.com
- Cloudflare docs, Set a Pay Per Crawl price: developers.cloudflare.com
- Cloudflare, Your Content, Your Rules press release (1 July 2026): cloudflare.com
- Cloudflare, Have it both ways: stay discoverable in search while disallowing AI training (15 September 2026): blog.cloudflare.com
- Cloudflare, The crawl-to-click gap (29 August 2025): blog.cloudflare.com
- RFC 9110, HTTP Semantics: rfc-editor.org
- RFC 9421, HTTP Message Signatures: rfc-editor.org
- IETF Web Bot Auth working group: datatracker.ietf.org
- RSL standard, launch (10 September 2025) and 1.0 (10 December 2025): rslstandard.org, rslstandard.org
- Microsoft Advertising, Building toward a sustainable content economy (3 February 2026): about.ads.microsoft.com
- Fastly on TollBit’s Bot Paywall integration (17 July 2025): fastly.com
- Akamai on monetizing AI bots at the edge (17 September 2025): akamai.com
- TollBit, State of the Bots, Q3-Q4 2025 and H1 2026: tollbit.com, tollbit.com
- US Copyright Office, Copyright and Artificial Intelligence, Part 3, pre-publication version (9 May 2025): copyright.gov