Ask a proxy provider for its success rate and you will usually get a very high number. Ask how it was counted and the answer is often “requests that returned a response”, or “requests that returned HTTP 200”. Neither tells you whether you got the data.
Whatever residential proxies you run, the useful definition of success is narrower: the response contained the thing you asked for. This guide explains why status codes overstate success, how to replace them with a pass rule, and how to count the result honestly. It focuses on measurement. For spotting poisoned or fake content inside a scraper, see our post on detecting blocked or fake content.
Why the status code lies
HTTP status codes describe the conversation between your client and a server. They say nothing about whether that server gave you the page. Defended sites have every reason to answer a suspicious request politely:
- Challenge pages. A CAPTCHA or a JavaScript interstitial is served with a 200, because the site wants a browser to render it.
- Soft blocks. A search page with zero results, a product page with the price removed, or a “something went wrong” template, all with a 200.
- Consent and redirect pages. A cookie wall or a regional landing page that replaces the content you requested.
- Wrong content. A page for a different country than the one you targeted, which parses fine and is wrong.
Each of these is billed as traffic, and each counts as a success on a status code dashboard. That is how status code success rates sit close to perfect while the share of usable pages is much lower.
Four outcomes, one success
A more honest scheme, the one the Atlas methodology uses, sorts every request into one of four outcomes:
- Pass. The response met the written condition for that target: the element parsed, the JSON had the expected shape, or the token was issued. This is the only success.
- Challenge. A CAPTCHA or interstitial came back instead of the page, usually as a 200.
- Block. The target refused outright: a 403, a 429, a block page or a connection reset before any content.
- Timeout. No usable response inside the time budget for that target, including errors raised by the proxy gateway itself.
Splitting failures this way matters because each one points at a different cause. Blocks and challenges mostly describe how the target sees the exit address. Timeouts mostly describe the network. A single failure count hides which problem you have.
Writing a pass rule
A pass rule is a check that code can run on a response without a human looking at it. It names the evidence that the page is the real one. Good rules are specific to the target:
- Product page: the price and stock markup are present and parse.
- Search results page: at least a set number of organic result blocks parse.
- JSON endpoint: the response is valid JSON with the expected array, not an error object.
- Queue or session: a token was issued.
- Static page: the body arrived unmodified, for example by matching a known hash or length.
Keep each rule narrow enough that a challenge page cannot pass it by accident, and loose enough that a legitimate layout change does not fail every request. Check for the data you need, not for the absence of the word “captcha”, which a new challenge vendor will not contain.
Write the rule down before you test. A rule chosen after looking at the results tends to be tuned to make one provider look good.
Challenge rate: the number status codes hide
Of the three failure types, the challenge rate deserves its own line in any report. It is the share of requests answered with a CAPTCHA or interstitial, and it behaves differently from blocks.
Sites tend to escalate: first they challenge an address, then they block it. A rising challenge rate is often the first sign that a pool is being used too hard or that a target has tightened its defences, days before hard blocks climb. Because challenges arrive as 200s, a status code dashboard will show nothing at all while this happens.
Challenges also cost money. A challenge page is a full response with headers, markup and scripts, so on a per-gigabyte plan you pay for it like a real page. The true cost page of Atlas shows how much that adds once failed requests are priced in.
Counting it fairly
A pass rule fixes the numerator. The denominator needs the same care:
- Count every attempt. If your client retries, each retry is a request. A success rate measured after retries describes your retry logic, not the network.
- Keep timeouts and gateway errors in. Dropping requests that “never reached the site” removes exactly the failures the network is responsible for.
- Weight time evenly. A busy hour should not drown out a quiet one, or a good afternoon hides a bad night.
- Report per target. An average across an undefended page and a defended one describes neither.
- Publish the denominator. A rate without its request count cannot be judged.
Doing it yourself
A minimal setup for measuring proxy success rate on your own work:
- List the targets you actually pay proxies for.
- Write one pass rule per target before sending a request.
- Log every request with its status, the verdict of the pass rule, the timing and the exit address the target saw.
- Classify each failure as challenge, block or timeout.
- Report success, challenge rate, block rate and timeouts separately, per target, with counts.
- Watch the challenge rate over time, and alert on it rather than on status codes.
Our post on monitoring a scraping pipeline covers the alerting side once these numbers exist.
Atlas judges every request it sends the same way, with a written pass rule per target, and publishes the result for each one on the Targets page, so a network’s success rate always means the share of requests that delivered the real page.