A site almost never goes from serving your crawler normally to blocking it outright in a single step. What happens first is quieter. A few more requests hit a challenge page. Some responses come back 200 OK with nothing useful inside. Latency creeps up. A handful of records fail validation. Each signal on its own looks like noise, and each one lives on a different dashboard, so nobody connects them until the dataset has a hole in it.
A target health score is the fix: one number per site that says whether this site is still behaving like its own normal, with the reasons attached. This essay covers which signals feed it, how to combine them without fooling yourself, and what the crawler should do when the number falls.
Target health is not proxy health
It is worth being precise about what is being measured, because the two get confused.
Proxy health asks whether a route works: a gateway, a country, a session, an exit. Target health asks whether a specific site is still willing and able to serve you. A dead proxy route fails against every site. A site turning against you fails across every route.
That difference is the first diagnostic. When failures rise, split them by route. If they cluster on one country, one pool or one session pattern, it is a route problem, and the approach in monitoring residential proxy health at scale applies. If they rise evenly across every route you send to that site, the site changed its mind about you, and that is what this score is for.
The signals
Group the signals by what they reveal. Some are loud, some are nearly silent, and the silent ones are the most expensive to miss.
| Signal | What it looks like | Why it matters |
|---|---|---|
| Hard blocks | 403, 429, connection resets | Explicit refusal; easy to count |
| Challenges | CAPTCHA or interstitial pages | The site suspects automation and is testing |
| Soft blocks | 200 OK with a block page, empty results or a stripped layout | Refusal disguised as success |
| Latency drift | p95 response time rising against the site’s own normal | Often deliberate slowing, sometimes just load |
| Extraction failures | Required fields missing, schema no longer matching | The page changed, or you are being served something different |
| Cost drift | More retries, bytes or credits per valid record | Everything above, expressed as money |
Soft blocks deserve special attention because status codes lie. In a 2023 measurement of geo-blocking from Cuba, 32 domains served their block pages with a 200 OK status, as described in which countries get geo-blocked most. A crawler that trusts the status code records those as successes. Detect soft blocks by content: known block-page markers, response size far below the page type’s normal, or an extraction that returns nothing where it always returned something.
Two owners, two scores
One distinction saves a great deal of confusion: separate “the site changed” from “the site turned against you”.
A redesign breaks your parser. Every page loads fine, but required fields vanish. That is an extraction problem with a code fix, owned by whoever maintains the parser. A site that starts challenging and soft-blocking is a relationship problem, and the fix is a change in how, how much or whether you collect. Different owners, different responses.
So compute hostility, meaning blocks, challenges, soft blocks and latency, as the health score, and track extraction validity alongside it as a separate parser-health signal. When both drop at once, look at hostility first: a site serving challenge pages will also break every extractor.
Score against the site’s own normal
The most common mistake is to use global thresholds. A 5% challenge rate is alarming on a site that has never challenged you, and entirely normal on one that challenges everybody on the first visit. Every signal has to be compared with that site’s own baseline, over a window long enough to be stable, such as the previous two weeks, and excluding the most recent day, so a bad day does not become the new normal.
Then weight, cap and sum:
from dataclasses import dataclass
@dataclass
class Window:
requests: int
hard_blocks: int # 403, 429, connection resets
challenges: int # CAPTCHA or interstitial pages
soft_blocks: int # 200 OK carrying a block page or an empty result
p95_latency_ms: float
MIN_REQUESTS = 50
WEIGHTS = {"hard_block": 35, "challenge": 25, "soft_block": 25, "latency": 15}
def signals(w):
n = max(1, w.requests)
return {
"hard_block": w.hard_blocks / n,
"challenge": w.challenges / n,
"soft_block": w.soft_blocks / n,
"latency": w.p95_latency_ms,
}
def badness(name, now, base):
if name == "latency":
# Full penalty at three times the site's own normal latency.
return min(1.0, max(0.0, (now / max(base, 1.0) - 1) / 2))
# Full penalty at 20 percentage points above the site's own normal rate.
return min(1.0, max(0.0, (now - base) / 0.20))
def health_score(current, baseline):
if current.requests < MIN_REQUESTS:
return None, ["not enough requests to judge"]
now, base = signals(current), signals(baseline)
penalty = {k: w * badness(k, now[k], base[k]) for k, w in WEIGHTS.items()}
score = round(100 - sum(penalty.values()))
reasons = [k for k, p in sorted(penalty.items(), key=lambda kv: -kv[1]) if p >= 1]
return score, reasonsA few design choices here are deliberate.
- Only excess over baseline counts. A site that has always challenged 3% of requests loses no points for doing so today.
- Each signal is capped. One runaway signal cannot push the score below zero or drown out the others, and the reasons list shows which one dominated.
- A small window returns no score. Five requests with one block is not a 20% block rate, it is not enough data. An absent score is more honest than a confident wrong one.
- The weights are opinions. Start with something like these, then adjust after reviewing a few real incidents. Hard blocks and soft blocks deserve the most weight because they mean data you did not get.
Smooth the score over time too, for example with an exponentially weighted average, so a single bad minute does not flap an alert, while a steady decline still shows up within the hour.
Canaries: ground truth you control
Every signal above is inferred. Canaries give you something closer to truth. Pick a handful of stable pages per site where you know what the right answer looks like: a product whose price you can check, a listing whose item count you know, a page whose structure has not changed in months. Fetch them on a schedule through the same path as production traffic.
When a canary returns a page that looks normal but carries the wrong value, you have found the hardest failure to detect: content that parses perfectly and is simply not what a real visitor would see. No status code or latency graph will show you that. Canaries also make the baseline trustworthy, because you know their correct behaviour independently of the crawler.
Tie actions to bands, and keep humans in the loop
A score nobody acts on is a dashboard. Give it bands, and give each band an action the system takes on its own:
| Band | Meaning | Automatic action |
|---|---|---|
| 80 to 100 | Behaving like its own normal | None |
| 50 to 79 | Degrading | Reduce concurrency for this site, lengthen revisit intervals, check whether failures are site-wide or route-specific |
| Below 50 | The site is clearly pushing back | Pause the site, keep only canaries running, alert a person |
The degrading band feeds straight into the rest of the crawler. Lower concurrency is what the per-host limiter in backpressure and flow control is for, and a falling score should raise the effective cost of that site in a cost-aware scheduler, so budget moves to sites where it still buys data.
The bottom band is deliberately not automated beyond a pause. A site that is strongly pushing back is telling you something, and the right response is a human decision: slow down further, check whether your collection still fits the site’s terms, look for an official API or data feed, or stop. Escalating automatically to heavier collection methods whenever the score drops only turns a signal into an arms race, and it is exactly what a health score should help you avoid. Rate limiting and request throttling covers how to read what a site is asking of you.
Review it like any other alert
Treat the score as an alert with a false-positive rate, and review it. After each incident, ask whether the score moved early enough, whether the reasons pointed at the real cause, and whether the automatic action helped. Most tuning comes from two or three real incidents, not from designing weights in advance. Keep the history: the score over months is the best record you have of how each site’s attitude to automated traffic changes.
The bottom line
Sites turn against crawlers gradually and quietly, through challenges, disguised refusals and slower answers, long before an outright block. Each signal alone is ambiguous. Compared against the site’s own normal, weighted, capped and combined, they give one number that moves early, with reasons attached.
The score is most valuable for what it lets the crawler do calmly: ease off before being blocked, move budget elsewhere, and hand the hard decisions to a person with the evidence in front of them. The broader metrics around it are in monitoring a web scraping pipeline.