Ask a scraping team how things are going and you will usually get one number: success rate. It is the obvious metric and it is genuinely useful, but on its own it is close to useless for running a pipeline, because it hides the two questions a team lead actually needs answered. Is the data we are collecting correct, and is what we are paying to collect it reasonable?
A pipeline can report ninety-eight percent success while quietly filling a warehouse with challenge pages, and it can report the same number this month as last while costing twice as much. Here is the metric set that catches both, organised by what each one is for.
Start with the metric everyone gets wrong
Validated success rate. Not the share of requests that returned 200, but the share that returned the data you wanted.
This distinction is the single highest-value change most teams can make to their monitoring. A challenge page, an empty result set, a truncated listing, a consent wall, or a generic regional redirect can all arrive with a 200 status. A counter that trusts status codes will report health while your dataset degrades, and you will find out from a business user asking why a chart looks wrong rather than from your own dashboard. Define a per-target validity check, an expected element, a plausible content length, a required field in the JSON, and count only responses that pass it, as described in detecting blocked or fake content.
Track it per target and per region, never as a single global figure. A global 95% could be every target at 95%, or nineteen targets at 100% and one at zero, and those call for completely different responses.
Data quality metrics
Beyond validity, three numbers tell you whether the output is trustworthy.
Coverage. Of the records you expected in a run, how many did you actually get? A job that succeeds on every request it makes, but makes fewer requests than yesterday because pagination broke or a discovery step returned less, will look perfect on success rate and be materially incomplete. Coverage catches that.
Freshness. How old is the newest data per source? For fast-moving work such as pricing or availability, staleness is a defect even when everything technically succeeded, so track the age of the most recent successful collection per target and alert when it exceeds what the use case tolerates.
Field-level completeness. Of the records collected, what share have every field you need? A layout change that removes one attribute rarely fails a request; it just quietly nulls a column, and only field-level tracking surfaces it early.
Together these answer “is the data correct”, which success rate alone cannot.
Efficiency and cost metrics
These answer “is what we are paying reasonable”, and they are usually the least instrumented.
Bytes per record. The most useful cost metric in the whole set. Divide bandwidth consumed by useful records extracted, per target. It normalises across differently sized jobs, it makes targets comparable, and a sudden rise is almost always someone rendering a full page where a data endpoint would do. Since residential proxies bill by data transferred, this number is close to your unit cost, and the levers on it are in cutting proxy bandwidth costs and choosing when a headless browser is warranted.
Cost per thousand records. Bytes per record multiplied by your rate, expressed per target. This is the number to put in front of anyone asking whether a data source is worth collecting, because it converts infrastructure into the language the business already uses, and it is what makes forecasting in estimating monthly bandwidth concrete.
Retry ratio. Retries as a share of total requests, per target. This is a leading indicator: it rises before success rate falls, because a pipeline that retries its way to a normal-looking result is hiding a problem rather than solving it. It is also pure waste on a bandwidth-billed product, so it is simultaneously a health signal and a cost line, which is why it sits at the centre of retry and backoff.
Bandwidth per target. Where the money actually goes. Teams are routinely surprised to find one target consuming the majority of a plan, and the fix is usually cheap once it is visible.
Performance metrics
Latency percentiles, not averages. Track p50, p95, and p99 per target. Residential connections are slower than direct ones by nature, so the absolute number matters less than the shape and the trend. An average hides the tail entirely, and the tail is what determines whether a time-boxed job finishes.
Throughput against plan. Records per hour, compared to what the schedule requires. This is how you know whether you need more concurrency, better pacing, or a different collection strategy, and it is the practical input to the distribution question in how many proxy IPs you actually need.
Block and challenge rate. Distinct from failed requests: the share of responses that were specifically a block, a captcha, or a challenge. Rising challenge rate with stable success rate means you are working harder for the same output and is an early warning that a target’s defences changed.
What not to track
Two things get measured a lot and deserve less attention than they receive.
Pool size. It is a vendor number, not a performance metric, and it does not predict your results. Density in the countries you actually collect from does, and the only way to know that is your own success rate per region.
Raw request count. Volume without validity is activity, not output. A pipeline that doubles its requests while collecting the same number of usable records has got worse, and a request counter will call that growth.
Similarly, be careful about how you use an uptime SLA. Provider availability is real and worth watching, but it is a different axis from whether your targets accept your traffic, and confusing the two leaves you contractually covered and operationally blind, which is the boundary discussed in proxy SLAs and uptime guarantees.
Turning metrics into alerts
A dashboard nobody looks at is decoration. A small number of alerts, tied to the metrics above, is what actually protects a pipeline.
Alert on validated success rate falling below a per-target threshold, since one number cannot fit every site. Alert on coverage dropping relative to the same run last week, which catches silent shrinkage. Alert on freshness exceeding the tolerance of the use case. Alert on bytes per record rising sharply, which is a cost regression and usually a code change. And alert on retry ratio climbing, because it fires earliest of all.
Two practical rules make these liveable. Compare against a rolling baseline rather than a fixed number, because targets have their own rhythms and a static threshold either cries wolf or never fires. And require a sustained deviation rather than a single bad interval, since one bad five-minute window is noise. The instrumentation itself is covered in monitoring a web scraping pipeline.
A minimal starting set
If you are instrumenting from scratch, six metrics per target get you most of the value: validated success rate, coverage against expected, freshness of the latest record, bytes per record, retry ratio, and p95 latency. Add region as a dimension on the first two, since geography is where multi-market pipelines fail silently. Everything else can wait until one of those six raises a question you cannot answer.
The bottom line
Success rate answers whether requests completed, which is the least interesting of the three questions a team lead has. Validate what came back before counting it as success, then measure coverage, freshness, and field completeness to know whether the data is correct, and bytes per record, cost per thousand records, and retry ratio to know whether the price is reasonable. Track everything per target and per region rather than globally, alert on sustained deviation from a rolling baseline, and resist judging infrastructure by pool size or raw request volume. The metrics that matter are the ones that change a decision, and these are the ones that do.
Measuring against the network itself is a separate exercise, covered in testing speed, success rate, and location accuracy. The residential proxies underneath are billed per GB, which is exactly why bytes per record is the cost metric worth watching: it is the number your invoice is made of.