Residential Proxy Retry Logic: Handling Failed and Rate-Limited Requests

Every scraper needs retry logic, and most of them grow it accidentally: a try block here, a time.sleep there, an attempt counter added the week something broke. The result works until the day a target has a bad hour, at which point the retry path does more damage than the original failure did. The argument for why that happens is in retry and backoff; this is the implementation.

The organising idea is that a retry is a decision made from a classified failure, not a loop wrapped around a request. Get the classification right and the rest follows.

Classify first

Every failure falls into one of four classes, and each has exactly one correct response.

Class	Examples	Response
Transient	timeout, connection reset, 502, 503, 504	retry with backoff
Rate	429, Retry-After present	wait as instructed, same route
Identity	challenge page, persistent 403, block page	new session, then retry
Terminal	404, 400, 401, 407, parse failure	do not retry, surface it

Two of these are the ones people get wrong. Rate signals are instructions about pacing, so the response is to wait rather than to swap addresses, because rotating in order to sustain the same pace is exactly the behaviour that escalates a throttle into a block. Terminal failures must never be retried: a 407 means credentials or a malformed username flag and will be equally wrong on attempt five, and a parse failure is a bug in your code that a retry reproduces faithfully forever.

The fifth case is the one that never appears in a status code at all: a response that looks fine and is not. A challenge page, an empty result set, or a truncated listing served with a 200 belongs in the identity class, but only if you notice it, which means validating the body before you decide anything. That check is the foundation of the whole system, per detecting blocked or fake content.

Classification in code

import random, time, requests

TRANSIENT = {502, 503, 504}

def classify(resp, exc, is_valid):
    if exc is not None:
        return "transient"                      # timeout, reset, DNS
    if resp.status_code == 429:
        return "rate"
    if resp.status_code in TRANSIENT:
        return "transient"
    if resp.status_code in (403, 401) or looks_like_challenge(resp):
        return "identity"
    if resp.status_code == 200 and not is_valid(resp.text):
        return "identity"                       # soft block: 200 but not our data
    if resp.ok:
        return "ok"
    return "terminal"                           # 404, 400, 407, everything else

looks_like_challenge is per target and usually a short list of markers: a captcha script reference, a known interstitial title, a body far shorter than a real page. Keep it in one place per target so both the retry path and your monitoring use the same definition.

Backoff with jitter, and honouring Retry-After

For the transient class, the delay grows exponentially and must be randomised. Fixed delays synchronise your workers, so a hundred requests that fail together retry together, and the burst survives the backoff.

def backoff(attempt, base=1.0, cap=60.0):
    ceiling = min(cap, base * (2 ** attempt))
    return random.uniform(0, ceiling)           # full jitter

For the rate class, the target may tell you exactly how long to wait, and that instruction beats your own schedule:

def wait_for(resp, attempt):
    ra = resp.headers.get("Retry-After")
    if ra:
        try:
            return min(float(ra), 300)          # honour it, but cap it
        except ValueError:
            pass                                 # HTTP-date form, fall through
    return backoff(attempt, base=2.0)            # rate signals start slower

Putting it together

MAX_ATTEMPTS = 4

def fetch(url, country, is_valid, session_id=None):
    sid = session_id
    for attempt in range(MAX_ATTEMPTS):
        proxies = build_proxies(country, sid)     # sid=None means rotate
        resp = exc = None
        try:
            resp = requests.get(url, proxies=proxies, timeout=20)
        except requests.RequestException as e:
            exc = e

        kind = classify(resp, exc, is_valid)

        if kind == "ok":
            return resp
        if kind == "terminal":
            raise TerminalError(url, getattr(resp, "status_code", None))
        if kind == "identity":
            sid = new_session_id() if sid else None   # retire the session
            time.sleep(backoff(attempt))
        elif kind == "rate":
            time.sleep(wait_for(resp, attempt))       # wait, do not rotate
        else:
            time.sleep(backoff(attempt))

    raise Exhausted(url)

Three details in there matter more than the structure. The function raises on terminal errors rather than returning None, so a caller cannot mistake a failure for empty data. Identity failures replace the session rather than reusing it, because the previous one is already known to the target. And rate failures deliberately do not touch the session, keeping the same route while slowing down.

Guardrails that stop retries becoming the problem

Per-request logic is not enough on its own, because it has no view of the system. Three additions do most of the protective work.

A retry budget caps retries as a share of total traffic to a target, say ten percent. In normal operation you never approach it. When a target breaks broadly, the budget is exhausted immediately and the extra retries simply do not happen, which is the behaviour you want, since retries help with isolated failures and actively hurt during a general outage.

A circuit breaker per target stops sending entirely once the failure rate crosses a threshold, waits out a cooldown, then lets a trickle through to test recovery. This protects the target from your pile-on and protects your addresses from accumulating failures against a site that is not answering anyone.

A shared rate limiter that retries also pass through. If retries bypass your pacing, your error path becomes an unthrottled flood exactly when the target is least able to take it. Route every attempt through the same limiter, per rate limiting and throttling.

All three belong in the component that already sees every request, which is the argument for a proxy manager rather than per-job retry code.

Idempotency and the dead-letter queue

Two practical concerns that are easy to forget.

Retries assume the operation is safe to repeat. For collection that is almost always true, since fetching a page twice costs bandwidth and nothing else. If any part of your pipeline writes as a side effect of fetching, make the write idempotent, keyed on something stable, so a retry does not create a duplicate record.

And when a request exhausts its attempts, do not discard it. Push it to a dead-letter queue and reprocess it much later, in the next run or after a long cooldown, rather than inside the current burst. Most items that fail during an incident succeed on their own an hour later, and a queue turns a hard failure into a deferred one at no cost.

Watch the retry rate

Retries are the earliest warning you get. Retry ratio, meaning retries as a share of requests per target, rises before success rate falls, because a pipeline that retries its way to a normal-looking outcome is hiding a problem rather than solving it. It is also pure cost on a bandwidth-billed product, so it is simultaneously a health metric and a spend metric. Put it on the dashboard next to validated success rate, per proxy KPIs and monitoring your pipeline.

The bottom line

Retry logic is a classifier followed by four responses, not a loop with a counter. Validate the body so soft blocks are classified as failures, then wait on rate signals without rotating, retire the session on identity failures, back off with jitter on transient ones, and never retry a terminal error. Cap attempts and delay. Then add the guardrails that a per-request view cannot provide: a retry budget so a broad outage cannot become a flood, a circuit breaker per target, and a shared limiter that retries also respect. Send exhausted requests to a dead-letter queue for a later run, and watch retry ratio as your earliest signal that something is changing.

The failover half of this, having somewhere clean to retry to, is what residential proxies provide: a large pool of real home-grade addresses so a retired session is replaced rather than reused, with per-GB pricing that makes disciplined retries directly cheaper than undisciplined ones.