A scraping pipeline rarely fails with a bang. The cron keeps firing, the process keeps exiting zero, the logs are full of 200 OK, and everything looks healthy right up until someone three teams away notices the numbers in a report went sideways a week ago. By then you have lost days of data and cannot re-collect most of it, because the pages have moved on.
This is the observability leg of running collection at scale. If Kubernetes gives you the orchestration and per-response validation catches the individual bad page, monitoring is the layer that watches the whole fleet over time and tells you a target turned on you before the corruption reaches your warehouse. Here are the metrics that matter, how to baseline them, and how to alert without drowning in noise.
Logs are not monitoring
The first thing to accept is that logs and exit codes tell you almost nothing about whether scraping is working. A process that exits zero and logs a 200 has told you it made a request and got a response, not that it got the right response. As covered in detecting blocked or fake content, a block page, a stripped shell, and real data all come back as 200. Monitoring means metrics: numbers emitted per request, aggregated over time, broken down by dimension, and compared against a baseline. That is a different discipline from logging, and it is the one that catches silent failure.
The metrics that matter
Emit these per request, tagged by target host and by exit geo, and aggregate them. None of it needs machine learning, just counters and histograms.
Success rate. The headline metric, and the one everyone defines wrong. Success is not HTTP 200, it is passed validation: the response contained the field or element a real page always has. Track the percentage of requests that pass your content assertion, per target host. A drop on one host while others hold steady is that host changing its behavior toward you, and it is the single most important number on your dashboard.
Block rate. The share of responses your detection layer flags as soft blocks or challenges. This is your early warning that a target got more aggressive or your pool reputation slipped. A creeping block rate is what precedes a hard ban, so watch its slope, not just its value, and correlate it with IP reputation and the blocks you are trying to avoid.
Latency percentiles. Track p50, p95, and p99, never the average, which hides the slow tail where the real problems live. A climbing p95 means a target is throttling you, a pool is degrading, or a route is congested. This is the fleet-level view of the same signal behind request timeouts and latency tuning.
Field fill rate. For each field you extract, the percentage of records where it actually populated. This is your data-quality canary: a field that was 98 percent filled yesterday and 20 percent today did not get rarer, you started receiving stripped or partial pages. Fill rate catches degradation that success rate can miss.
Throughput versus backlog. Requests and records per minute, watched against the depth of your work queue. Rising throughput with a shrinking backlog is healthy. Flat throughput with a growing backlog means you are falling behind and need to scale out, the signal that feeds autoscaling on queue depth.
Freshness. How old the newest record is for each source. A source whose freshness stops advancing has silently stopped producing, even if every other metric looks fine. This is the one that catches a feed that died without erroring.
Cost per record. Bandwidth and spend divided by usable records collected. Beyond efficiency, this is an anomaly detector: a sudden jump in bytes-per-record often means you are downloading block pages or bloated junk instead of data, so a health problem shows up as a cost spike first. It also keeps the bandwidth bill honest.
Retry rate and error taxonomy. Attempts per successful record, broken down by failure type: timeout, 429, connection reset, DNS, detected block. The shape of your errors is a diagnosis. A spike in 429s means you are pushing a target too hard; a spike in connection resets points at the network or proxy layer; a spike in detected blocks points at reputation. One aggregate error count tells you something is wrong; the taxonomy tells you what.
Baseline per target, alert on deviation
The mistake that makes monitoring useless is alerting on absolute thresholds. A 3 percent block rate is perfectly normal for one site and a five-alarm fire for another. A p95 of two seconds is fine for a heavy page and terrible for a light API. Absolute thresholds either miss real problems or cry wolf constantly, and alert fatigue means the one real page eventually gets ignored.
Baseline every metric per target host, then alert on deviation from that baseline: a sustained drop in success rate, a block rate climbing faster than its normal band, a p95 that doubled against last week. Rate-of-change and per-target deviation catch the things that matter and stay quiet when a site is simply different from another. Alert on sustained deviation rather than a single bad minute, and reserve paging a human for what actually needs one, everything else can be a dashboard or a digest.
# Emit per response; aggregate into a time series, dimensioned by host + geo.def record(metrics, host, geo, resp, validation): tags = {"host": host, "geo": geo} metrics.incr("requests", tags) metrics.incr("success" if validation.ok else "failure", tags) if validation.blocked: metrics.incr("blocked", tags) # block rate = blocked / requests metrics.observe("latency_ms", resp.elapsed_ms, tags) # histogram -> p50/p95/p99 metrics.observe("bytes", resp.size, tags) # -> cost/bytes per record for field, present in validation.fields.items(): metrics.incr(f"field.{field}." + ("filled" if present else "empty"), tags)Canaries and the proxy dimension
Two things sharpen all of the above. First, run canaries: fetch a page whose correct content you already know, on a schedule and per geo, and assert it still matches. A canary flips from healthy to broken the instant a target changes, without waiting for a slow decline in aggregate metrics to become obvious, and doing it per geo catches a site that blocks one country’s exit IPs while leaving another untouched.
Second, dimension every metric by exit geo and pool, not just by target. Problems are frequently localized: one country’s IPs get challenged while the rest sail through, or one segment of a pool degrades. Without the geo and pool breakdown, that shows up as a mild, confusing dip in the global average instead of the sharp, actionable signal it actually is. Because pool quality is a leading indicator of block rate, watching success and block rate per pool tells you about a degrading residential pool before it drags down the whole run, and gives you the option to shift load or fail over before users notice.
The bottom line
A scraping pipeline that is not monitored is a pipeline that fails silently and expensively. Instrument success as passed-validation rather than HTTP 200, and track block rate, latency percentiles, field fill rate, throughput versus backlog, freshness, cost per record, and a proper error taxonomy, every one of them broken down by target host and exit geo. Baseline each metric per target and alert on deviation from that baseline, not on absolute numbers, so you catch the real breaks without drowning in false ones. Add canaries for instant early warning. Do that and the silent failure stops being silent: you find out in the pipeline, in minutes, instead of in a downstream report, in weeks.
Since block rate and latency both trace back to the quality of the IPs you exit through, a clean residential pool is what keeps those metrics healthy in the first place, and the per-GB pricing makes the cost-per-record you are now watching something you can actually optimize.