Every scraping dashboard has a success rate, and nearly every one of them counts the same thing: responses with an HTTP 200. It is an easy number to collect and a comforting one to report. It is also the most misleading number in web data collection, because a 200 only means a server sent something back. It says nothing about whether that something was the page you asked for.
The gap between those two things is the silent failure rate: the share of “successful” requests that returned the wrong content. It is the failure you cannot see in your logs, and it lands in your dataset looking exactly like good data.
So how big is it? The honest answer is that nobody has published it. That absence turns out to be the most interesting finding, and the rest of this piece covers why it exists, what the nearby measurements do show, and how to measure your own.
Key takeaways
- No published study measures what share of HTTP 200 responses carry the wrong content across the web. The most recent large bot-blocking study states in writing that content degradation within 200 responses was outside its scope.
- Blocking is common and poorly labelled. A study of Common Crawl found at least 1.68% of sites explicitly refusing the crawler, with an inconsistent and even incorrect use of status codes.
- Block pages do arrive as 200s. In a 2023 measurement from Cuba, 32 of the 395 domains that served a block page did so with a 200 status.
- Major link-rot studies count soft 404 pages as live, because checking content is much harder than checking status codes.
- Your own silent failure rate is measurable, but only by validating content, not status.
What counts as a silent failure
A silent failure is any response that reports success and does not contain what a real visitor would have received. The common forms:
| Form | What you receive | Why it passes as success |
|---|---|---|
| Block page served as 200 | A refusal message where the page should be | Status says OK |
| Challenge or interstitial | A CAPTCHA or “checking your browser” page | Often a 200, sometimes an error code |
| Soft 404 | A generic page or homepage for content that no longer exists | The server returns 200 instead of 404 |
| Empty shell | HTML with no content, because the page builds itself in JavaScript | A complete, valid document |
| Consent or pay wall | A consent screen in front of the content | The page loaded; the content did not |
| Wrong variant | Another country’s page, price or language | A perfectly real page, just not the one you meant |
Each of these parses, stores and aggregates like good data. None of them trips an error alert.
The number nobody has measured
Researchers who study bot blocking and web decay have repeatedly stepped around this question, and several say so explicitly.
The most recent large study, “Detecting Bot Detection” by Gundelach, Mühlhauser and Herrmann (University of Bamberg, June 2026), scanned the Tranco top 10,000 sites between 27 February and 2 March 2026 under several browser configurations. Its limitations section is direct: “content degradation within HTTP 200 responses is outside our observational scope.” The authors also surveyed 81 web measurement papers and found that “only 5% of papers explicitly quantify bot detection or blocking rates, and 83% omit any discussion” of it.
They are in good company. The PAM 2025 study of Common Crawl refusals worked from non-200 responses. The 2018 global geo-blocking study noted that a site might load while “the login button has disappeared, or that some content is not available”, and left those “more nuanced changes in content” to future work. Pew Research Center’s 2024 link-rot study treated as accessible “ambiguous situations in which we could not guarantee that the content exists, like soft 404 pages.” The Internet Archive’s April 2026 dead-web study “relied on HTTP status codes and did not look into the contents of the pages to check for any soft-404s.”
None of this is carelessness. Classifying status codes scales to millions of pages; judging whether content is correct requires knowing what correct looks like for each page. That is exactly why the silent failure rate is unmeasured at web scale, and exactly why it is measurable for your own targets, where you do know what correct looks like.
What the nearby measurements show
No single figure answers the question, but several measurements bound it.
| Finding | Source | Measured |
|---|---|---|
| Headless Chromium was soft-blocked on 15.2% of top sites, against 6.8% to 7.2% for other browser setups; 81.9% of soft-blocked sites were attributed to bot detection | Gundelach, Mühlhauser and Herrmann, arXiv, June 2026 | Feb to Mar 2026 |
| At least 1.68% of sites explicitly refused Common Crawl, with inconsistent and even incorrect status codes; 80% of refusing domains blocked every request | Ansar, Sperotto and Holz, PAM 2025 | Common Crawl snapshot, late 2023 |
| 32 of 395 domains serving block pages to Cuban users used a 200 status | Ablove et al., USENIX Security 2024 | May 2023 |
| Automated crawls missed 45% of the fingerprinting websites real users encountered, partly from failing to get past bot detection | Annamalai, Bilogrevic and De Cristofaro, WWW 2025 | 30 users over 10 weeks |
| Cookie walls on 0.6% of 45,000 sites, and on 8.5% of Germany’s top 1,000 | Rasaii, Gosain and Gasser, IMC 2023 | 2023 |
| 7.35% of web servers returned 200 for an unknown document instead of 404 | Prieto Álvarez, Álvarez Díaz and Cacheda Seijo, 2014 | Before 2014 |
| Soft 404s accounted for more than 15% of dead links | Bar-Yossef, Broder, Kumar and Tomkins, WWW 2004 | Before 2004 |
The first two rows describe blocking visible through error codes, which is the part that is easy to see. The third shows the other part exists: roughly one in twelve block pages in that study arrived dressed as success. The soft 404 figures are old, and they are the most recent that have been published.
What it costs downstream
The clearest picture of silent failure in a real dataset comes from official statistics. When the UK Office for National Statistics piloted price indices from web-scraped supermarket data, its May 2016 update reported that “the total percentage of products that were classified as anomalous or misclassifications after this validation step was 25%,” removing prices “from 3.4 million to 2.5 million.” Missing data, it added, “were mainly caused by retailers making structural changes to their websites.”
That 25% is not an HTTP-level failure rate. Most of it was products scraped into the wrong category and outlier prices. That is precisely the point: every one of those records came back from a request that succeeded, and a quarter of them were unusable. It took a validation step, built by a statistics office, to find them.
Why status codes cannot carry this signal
It would be convenient if servers reported refusals honestly. The evidence says they do not do it consistently. The Common Crawl study found refusals signalled through an inconsistent and even incorrect use of HTTP status codes. The Cuba study found blocks spread across DNS failures, timeouts, 403s, a handful of the dedicated 451 code, and 200s.
Some infrastructure does help. Cloudflare sets a cf-mitigated: challenge response header for all of its challenge page types, which is a far more reliable signal than the status code. Check for it. But a header from one provider is not a web standard, and most silent failures carry no marker at all.
Measuring your own silent failure rate
The definition is simple: of the responses your system counted as successful, the share that failed content validation. The work is in the validation.
- Validate records, not responses. Decide which fields every record of a page type must contain, and fail any 200 that does not produce them.
- Compare size against the page type’s normal. A product page that is suddenly a fifth of its usual size is rarely a product page.
- Look for block and challenge markers, including headers like
cf-mitigated, and phrases your targets actually use. - Run canaries. Fetch pages whose correct content you know independently, through the same path as production, and compare.
- Record where you fetched from. A wrong-country variant is only detectable if you logged the vantage point, which is the case made in the vantage-point standard.
- Sample for human review. A few dozen responses a week, read by a person, catches failure modes no rule anticipated.
A first-pass classifier can be very small:
BLOCK_MARKERS = ("captcha", "access denied", "unusual traffic", "verify you are human")
REQUIRED_FIELDS = ("title", "price")
def classify(resp, record, baseline_bytes):
"""Label one response. Anything but "ok" on a 200 is a silent failure."""
if resp.headers.get("cf-mitigated") == "challenge":
return "challenge"
if resp.status_code != 200:
return "http_error"
if not record and any(m in resp.text.lower() for m in BLOCK_MARKERS):
return "block_page"
if len(resp.content) < 0.2 * baseline_bytes:
return "too_small"
if not record or any(record.get(f) in (None, "") for f in REQUIRED_FIELDS):
return "missing_fields"
return "ok"Report the result per target and per page type, next to the success rate you already have. When the two diverge, the success rate is lying to you. Rising soft blocks on one site are also one of the earliest signs it is turning against your crawler, which is what a target health score is built to catch. The wider metrics sit in monitoring a web scraping pipeline.
Where tooling helps, and where it cannot
Managed collection removes some silent failures before they reach you. Shifter’s Web Scraping API retries failed fetches, CAPTCHAs and transient target errors automatically, up to three times with different proxies, and charges only for successful requests. JavaScript rendering removes the empty-shell problem for pages that build themselves in the browser, and extract_rules returns named fields, which makes a missing field easy to detect.
What no collection layer can do is know that a perfectly well-formed page contains the wrong price or the wrong country’s catalogue. Only you know what correct looks like for your data. Content validation belongs in your pipeline regardless of how the pages were fetched.
The bottom line
A 200 is a claim made by a server, not a guarantee about content. Block pages, challenges, soft 404s, empty shells, consent walls and wrong variants all arrive dressed as success, and published research, for sound practical reasons, has measured almost everything about web blocking except this.
The consequence is that the only silent failure rate you will ever have is the one you measure yourself. Validate every record, keep canaries whose answers you know, and put the result on the same dashboard as your success rate. The difference between the two numbers is the part of your dataset you currently cannot trust.
Sources and references
- Gundelach, Mühlhauser and Herrmann, Detecting Bot Detection: Prevalence, Techniques, and Implications for Web Measurement Research, arXiv, 12 June 2026. Scans 27 February to 2 March 2026.
- Ansar, Sperotto and Holz, Web Crawl Refusals: Insights From Common Crawl, PAM 2025, 7 March 2025.
- Ablove et al., Digital Discrimination of Users in Sanctioned States: The Case of the Cuba Embargo, USENIX Security 2024. Measured May 2023.
- McDonald et al., 403 Forbidden: A Global View of CDN Geoblocking, ACM IMC 2018.
- Annamalai, Bilogrevic and De Cristofaro, Beyond the Crawl: Unmasking Browser Fingerprinting in Real User Interactions, WWW 2025.
- Rasaii, Gosain and Gasser, Thou Shalt Not Reject: Analyzing Accept-Or-Pay Cookie Banners on the Web, ACM IMC 2023.
- Prieto Álvarez, Álvarez Díaz and Cacheda Seijo, Soft-404 Pages, a Crawling Problem, Journal of Digital Information Management, 2014.
- Bar-Yossef, Broder, Kumar and Tomkins, Sic Transit Gloria Telae: Towards an Understanding of the Web’s Decay, WWW 2004.
- Pew Research Center, When Online Content Disappears: methodology, 17 May 2024.
- Sawood Alam, Internet Archive, Gone but Not Forgotten: Recovering the Dead Web, 23 April 2026.
- Office for National Statistics, Research indices using web scraped data: May 2016 update, 23 May 2016.
- Cloudflare, Detect a Challenge Page response.
- Shifter, Web Scraping API errors and limits.