Ask how much of the web now sits behind a bot wall and you will hear big numbers. Most of them answer a different question. For teams that collect web data, the difference matters, because a site that could block you and a site that does block you are not the same problem.
No one has run a census, and nobody could: the answer depends on the page, the client and the moment. What exists is a set of measurements from different angles. Put together, they give a clearer answer than any single headline.
Three different questions
| Question | Measure | Population |
|---|---|---|
| Could the site block you? | 33.1% use a monitored reverse proxy, 28.5% Cloudflare | All websites (W3Techs, 5 Nov 2026) |
| 71% use a CDN, falling to 35% | Top 1,000 and top 10 million sites (Web Almanac 2025) | |
| Has it installed bot protection? | 31.1% on mobile, over 34% on desktop | Sites in HTTP Archive (Web Almanac 2025) |
| 16.1% running bot management we could detect | 653 top-site homepages (our test) | |
| Does it actually block? | 6.8% to 15.2%, by browser | Top 10,000 homepages (Feb and Mar 2026) |
| 5.3% to 8.1%, across eight crawlers | 1,902 popular landing pages (Nov 2026 preprint) | |
| 9.0% blocked, 9.8% challenged | 653 top-site homepages (our test) |
Read from top to bottom, the numbers shrink. About a third of websites could challenge a bot, but only about one in ten popular homepages turns a given automated client away.
These populations are different, so they do not form a neat funnel, and the top-site figures do not describe the long tail of the web. Even so, the direction is consistent across every source.
Capability is not enforcement
The most common mistake is reading a CDN’s market share as a block rate. A study of the top 10,000 sites, published in June 2026 and presented at ARES 2026, is the clearest test of this. It says “hosting on a capable provider does not guarantee active detection”, and it found blocks on 37% of Cloudflare-hosted sites in at least one test condition. So most of those sites did not block any of its browsers.
Our own test in October 2026 points the same way. Across the same 653 top-site homepages, vendor cookies or headers showed a bot-protection or edge-security vendor on 36.9%, but an active bot-management product on 16.1%. The bot protection figure from HTTP Archive also needs care, since around half of that category is reCAPTCHA, which is often a widget on a form rather than a wall around a site.
What a homepage test misses
Those block rates are probably low, for three reasons.
Challenges can arrive as a 200. The top 10,000 study counted only 403, 429 and 503 responses as blocks, and the November preprint counted 403 and 429. A challenge page returned with a 200 is not counted, so true interference is higher. Our test did count challenges, and they added about as much again as blocks.
Blocking grows past the homepage. The November preprint found that among sites with at least six pages per crawler, Chromium’s denial rate rose from 0.2% on landing pages to 10.4% on pages 2 to 6. Other clients rose less, and some did not rise at all. A homepage check flatters a site that only reacts to behaviour.
The client matters. The top 10,000 study found a 15% block rate for headless Chromium against about 7% for the other configurations. In our test, moving from a Chrome User-Agent to a library’s default raised blocks from 9.0% to 15.5%. Honest bot identification and sloppy configuration both cost access.
Is it growing?
Probably, though there is no simple series. The best long-run signal is Common Crawl, which publishes fetch results for every crawl. Its share of fetches recorded as denied, which includes robots.txt as well as 403s, looks like this:
| Crawl | Denied | Success |
|---|---|---|
| November 2019 | 3.07% | 76.94% |
| December 2022 | 2.94% | 74.79% |
| December 2025 | 5.80% | 63.90% |
| October 2026 | 4.85% | 62.82% |
Common Crawl attributes the recent decline in success to more sites disallowing crawling or denying it with a 403, and to a backoff it introduced in 2022, which skips more URLs. So the fall is not all blocking. These are shares of its fetches rather than of websites, and they reflect a crawler that identifies itself and honours robots.txt.
The pressure behind it is real. Thales’ 2026 Bad Bot Report puts bots at over 53% of all traffic in 2025, and Cloudflare reported that non-AI bots generated 44% of HTML requests as of December 2025. Each comes from a vendor’s own network, and both sell bot defences, but they explain why more site owners are switching these tools on.
What this means for your project
- Do not plan from a headline. Neither “a third of the web” nor “one in ten” describes your targets. In the top 10,000 study, blocking was “provider-driven, not rank-driven”, so two similar sites can behave very differently.
- Test your own list. Our anti-bot stack lookup shows how much one request can reveal about a site.
- Test past the homepage. Walk the pages you actually need, at the pace you will really run.
- Check content, not status. Silent failures that return a 200 do more damage than visible blocks.
- Fix your client before anything else. Identify yourself properly, stay at a sensible rate and keep sessions coherent.
Where a site treats your IP as the problem, for example because it comes from a datacenter range, residential proxies let you see public pages from a consumer connection. They do not change a site’s rules, and a site that has said no to automated access still means it.
The bottom line
The honest answer has three parts. About a third of websites could challenge a bot. Far fewer actively do, with roughly one popular homepage in ten turning away a given client. And the real figure is higher than any published block rate, because challenges and deeper pages are undercounted. For planning, that means most of the web is still open to a well-behaved client, the hard part is concentrated, and the only number that matters for your project is the one you measure on your own targets.
Sources and references
- W3Techs, Cloudflare usage statistics (5 November 2026): w3techs.com
- HTTP Archive, Web Almanac 2025, CDN chapter: almanac.httparchive.org
- HTTP Archive, Web Almanac 2025, Security chapter: almanac.httparchive.org
- Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research (June 2026): arxiv.org/abs/2606.14525
- Beyond the Landing Page: Website Blocking, Tracker Visibility, and Human Alignment in Automated Web Crawls (preprint, November 2026): arxiv.org/abs/2611.06197
- Common Crawl, crawler metrics: commoncrawl.github.io
- Thales, 2026 Bad Bot Report: cpl.thalesgroup.com
- Cloudflare Radar 2025 Year in Review: blog.cloudflare.com