A small scraping job either works or it does not, and you find out quickly. A large one is never in either state. At any moment some fraction of your traffic is failing, and the operationally useful question is not “is the proxy working” but “which slice of this has degraded, and is it us, the provider, or the target.”
Getting a useful answer means instrumenting along the dimensions that can fail independently, which is not the same as adding more metrics. Here is what to measure, how to probe without wasting bandwidth, and how to tell the three failure sources apart.
Health is per route, not per proxy
The first correction: on a pooled gateway you do not have proxies to monitor. You never chose the exit addresses, you do not keep them, and an address that failed once is not an entity you can track over time. What you can track is a route, meaning a combination of the things you control: target, country, and where relevant city or ASN.
So the unit of health is the route. amazon-de, serp-us-chicago, marketplace-jp. Each has its own success rate, latency profile, and failure mix, and each can degrade while the others stay perfect. A single global “proxy health” number averages exactly the signal you need away: ninety-five percent overall can be twenty routes at ninety-nine and one route at zero, and only the second reading tells you to do something.
Add session health as a second, shorter-lived unit. A sticky session that starts drawing challenges should be retired and replaced rather than reused, and that decision belongs in the same component that hands sessions out, as described in building a proxy manager.
Passive first: every real request is a health check
The cheapest monitoring is the traffic you are already sending. Every production request produces an outcome, and if you record it against its route you get continuous health data at no additional bandwidth cost.
The critical detail is what counts as success. Not a 200. A challenge page, a generic or empty result, a truncated listing, or a redirect to a landing page all return 200 and all mean your collection failed, so a monitor that counts status codes will report health while the dataset degrades. Validate the body against a per-target expectation before recording the outcome, which is the discipline in detecting blocked or fake content. This single change is what separates a health dashboard that catches problems from one that confirms your bias.
Record at minimum, per request: route, validated outcome, latency, bytes, and a failure class if it failed. That is enough to compute everything below.
Classify failures, because the class is the diagnosis
Counting failures tells you something is wrong. Classifying them tells you what. Five classes cover almost everything, and each points somewhere different.
Auth failures mean credentials or a malformed targeting flag, which is a configuration problem on your side and will not resolve on retry, per fixing 407 and credential errors. No-match failures, where the gateway has no address fitting your filter at that moment, mean your targeting is too narrow rather than that anything is broken; broaden from city to country and it clears. Rate signals, meaning 429 and friends, mean your pacing is too aggressive for that target, which is a throttling problem. Block signals, meaning challenges and persistent 403s, mean the target rejected the identity, so retire the session and consider whether your headers or fingerprint are the real cause. Transport failures, meaning timeouts and connection resets, are the ambiguous class and are worth their own investigation, since the reasons requests time out include target slowness, an unhealthy route, and your own concurrency being too high.
The mix matters more than the total. A route at eighty percent success made of rate signals needs slower pacing; the same route at eighty percent made of block signals needs a different identity strategy; made of no-match errors it needs broader targeting. Same number, three different fixes.
Active probes, used sparingly
Passive monitoring has one blind spot: it only covers routes you are currently using, so a route that is scheduled to run at 03:00 gives you no warning at 22:00 that it has broken. A small set of active probes fills that gap, but they cost bandwidth, so keep them cheap and purposeful.
Two kinds are worth running. A connectivity and geo probe per country, hitting a small endpoint that echoes the address and its location, confirms the gateway is reachable and that the country you request is the country you get. Keep it light, since this is the probe you run most often. A target canary per important target, fetching one known-stable page and validating it, tells you whether that specific target is answering normally, which is the check that distinguishes a target problem from a proxy problem.
import requests
def geo_probe(country):
proxy = f"http://customer-USERNAME-country-{country}:PASSWORD@p.shifter.io:443"
try:
r = requests.get("https://ipinfo.io/json",
proxies={"http": proxy, "https": proxy}, timeout=15)
d = r.json()
return {"ok": d.get("country","").lower() == country,
"got": d.get("country"), "org": d.get("org")}
except Exception as e:
return {"ok": False, "error": type(e).__name__}Run geo probes on a slow cadence across the countries you actually use, and canaries on a cadence matched to how much a silent failure would cost you. Alert on a probe failing repeatedly rather than once, since a single failure on a rotating pool is normal noise.
Telling the three failure sources apart
This is the question that actually gets asked during an incident, and the answer comes from comparing signals rather than from any single metric.
Your side shows as failures across many unrelated targets at once, usually starting exactly when something deployed. Auth failures everywhere, a spike in transport errors from one worker pool, or a rise in no-match errors after someone tightened a filter all point inward. The tell is breadth: your own bugs rarely respect target boundaries.
The provider shows as failures across many targets but confined to the network layer: connectivity probes failing, geo probes returning the wrong country, transport errors climbing while target canaries that do get through still return valid pages. This is also where a public status page earns its place, since correlating your own dip against the provider’s incident history answers the question immediately, and the tiering that governs what you are owed is in proxy SLAs and uptime guarantees.
The target shows as failures confined to one target while every other route stays healthy, and the canary for that target fails while its geo probes pass. If it fails from every region simultaneously the site is likely having its own problems; if it fails from one region only, you are looking at a geo-specific block or a regional edge issue.
Instrument so those comparisons are one query rather than an afternoon: the same failure classes and outcomes, tagged with route, region, and worker, is enough.
What to alert on
Dashboards are for investigating, alerts are for waking someone. Keep the alert set small and make each one actionable.
Alert on validated success rate for a route falling below its own rolling baseline, not a global threshold, because a route that normally runs at seventy percent against a hostile target is healthy at seventy percent and broken at forty. Alert on a shift in the failure mix, since a route holding its success rate while block signals replace rate signals has changed character in a way that predicts trouble. Alert on retry ratio climbing, which rises before success rate falls and is therefore the earliest warning you get. Alert on coverage, meaning a scheduled route producing materially fewer records than its own recent history, which catches silent shrinkage that success rate cannot see. And alert on a probe failing repeatedly for a region you depend on.
Require sustained deviation rather than a single interval, and compare against a rolling baseline. The metric definitions behind these are in proxy KPIs, and the pipeline-level instrumentation is in monitoring a web scraping pipeline.
Make the system act on it
Monitoring that only produces graphs leaves a human in the loop for problems a machine should handle. The same signals should drive automatic behaviour: retire a session that accumulates block signals, open a circuit breaker on a route whose success rate collapses so you stop feeding a target that is not answering, shift work to another region when one degrades, per failover in multi-region pipelines, and back off automatically when rate signals appear. Humans should be alerted for things that need judgment, not for things that need a rule.
The bottom line
At scale, health is not a property of the proxy, it is a property of each route you run, so instrument per target and region and let session health be its own short-lived signal. Judge every outcome on a validated body rather than a status code, because that is the difference between monitoring and self-deception. Classify failures, since the mix tells you whether to slow down, change identity, broaden targeting, or fix your own configuration. Add a thin layer of geo probes and target canaries to cover routes that are not currently running and to separate provider problems from target problems. Then alert on sustained deviation from each route’s own baseline, on failure-mix shifts, and on retry ratio, and wire the same signals into automatic retirement, backoff and failover so the system fixes what it can before anyone is woken up.
The layer underneath is residential proxies, where country and city targeting and session control are expressed per request, which is what makes route-level health a matter of tagging your own requests rather than an integration project, with per-GB pricing so a lean probe strategy costs almost nothing next to the collection itself.