A scraper hits a slow response, so it retries. The retry also fails, so it retries again, immediately, and so does every other worker that hit the same wall at the same moment. Within seconds you have sent a burst of traffic at a site that was already telling you it wanted less, and what started as a temporary throttle is now a hard block on every address involved. The site did not escalate. Your retry loop did.
This is the most common way a well-built scraper injures itself, and it is entirely avoidable. Retry logic is a load-shedding mechanism, not a persistence mechanism, and the difference between those two ideas is the difference between a job that degrades gracefully and one that gets itself banned. The broader hygiene is covered in avoiding blocks; this is the mechanics of the retry path itself.
Why naive retries make things worse
Three dynamics compound, and they compound together.
The first is that retries add load exactly when load is the problem. A 429 or a slow response is a request to send less, and answering it with more requests inverts the signal. The second is synchronization. Workers that fail at the same moment and wait the same fixed interval will retry at the same moment too, so instead of spreading out they arrive as a coordinated burst, and each round of that burst re-synchronizes the next one. The third is that retries are usually counted against you per address, so hammering a target from the same IP moves that address from throttled to flagged, which is a reputation problem that outlives the incident and follows the address into your next job.
Put together, a naive loop takes a recoverable condition and produces exactly the traffic shape that anti-bot systems are built to catch. The same failure, handled with restraint, would have resolved on its own.
Classify before you retry
The first rule is that not every failure deserves a retry, and the ones that do deserve different ones. Sort responses into three buckets.
Some failures are transient and worth retrying: connection resets, timeouts, 502, 503, 504, and 429. These represent a system that is momentarily unable rather than unwilling, and 429 in particular is an explicit instruction about pacing rather than a refusal. Some are terminal and must never be retried: 404, 400, 401, 403 that persists across addresses, and a page that parsed cleanly but contained nothing you wanted. Retrying these burns bandwidth and reputation for a result that cannot change, and a parse failure is a code bug that a retry will faithfully reproduce forever.
The third bucket is the dangerous one: responses that look successful and are not. A challenge page, a generic or empty result, a truncated listing, or a redirect to a landing page can all arrive with a 200 status, and a scraper that trusts status codes alone will happily record them as data. Validate the body before you count a response as a success, which is the substance of detecting blocked or fake content. A soft block is a retryable failure, but only if you notice it is a failure.
Exponential backoff, and why jitter is not optional
For the retryable bucket, the delay between attempts should grow, and the standard shape is exponential: wait one second, then two, then four, then eight. Growth matters because it gives a struggling target progressively more room instead of a constant drumbeat.
But exponential backoff alone is not enough, and this is the part people skip. If a hundred workers fail together and all wait exactly one second, they retry together one second later. The backoff grew, but the burst survived, and you have simply moved the same spike down the timeline. The fix is jitter: randomize each delay rather than using the computed value directly. Full jitter, meaning a random wait chosen between zero and the current ceiling, spreads a synchronized failure into a smooth distribution of retries. It is a two-line change and it is the single most effective thing in this article.
Two more rules go with it. Honor Retry-After when a site sends one, because that header is the target telling you exactly how long to wait and ignoring it in favor of your own schedule is both rude and counterproductive. And cap both the delay and the attempt count, because a request that has failed five times is not going to succeed on the sixth, and an unbounded retry is just a slow way of never giving up on something that is already lost.
import random, time
import requests
PROXY = "http://customer-USERNAME-country-us:PASSWORD@p.shifter.io:443"
PROXIES = {"http": PROXY, "https": PROXY}
TRANSIENT = {429, 502, 503, 504}
MAX_ATTEMPTS = 5
BASE, CAP = 1.0, 60.0
def fetch(url):
for attempt in range(MAX_ATTEMPTS):
try:
r = requests.get(url, proxies=PROXIES, timeout=20)
except requests.RequestException:
pass # transient: fall through to backoff
else:
if r.status_code == 200 and is_valid(r.text):
return r.text # validate the body, not just the code
if r.status_code not in TRANSIENT:
return None # terminal: do not retry
after = r.headers.get("Retry-After")
if after:
time.sleep(min(float(after), CAP)) # the target told you the answer
continue
ceiling = min(CAP, BASE * (2 ** attempt))
time.sleep(random.uniform(0, ceiling)) # full jitter, not a fixed delay
return NoneRotate or wait: the decision retries usually get wrong
With residential proxies there is a second axis. A failure can be answered with time, with a different address, or with both, and choosing wrong wastes one of them.
Wait when the signal is about rate. A 429 or a Retry-After is the target saying your pace is too high, and swapping to a fresh address so you can keep the same pace is precisely the behavior that looks like evasion and gets a whole pool burned rather than one route. Slow down instead.
Rotate when the signal is about the address. A block page, a persistent 403, or a challenge that keeps appearing on one route means that specific address is no longer trusted, and waiting will not restore it. Retire the route and continue on a fresh one, which is the failover pattern, and note that the reputation of the replacement is what determines whether the retry actually helps. The one case where rotation is wrong is mid-session work: if a flow depends on a held identity, changing address breaks it, so a failure inside a sticky session means restarting the sequence on a new session rather than swapping addresses underneath the existing one.
Timeouts sit between the two and are worth their own diagnosis rather than a reflex, since the reasons requests time out include target slowness, an unhealthy route, and your own concurrency being too high.
Budgets and circuit breakers
Per-request retry rules are not enough on their own, because they have no view of the system. Two mechanisms give you that view.
A retry budget caps retries as a proportion of total traffic, for example allowing retries to be at most ten percent of requests to a given target. Under normal conditions the budget is never touched. When something breaks broadly, the budget is exhausted immediately and the extra retries simply do not happen, which is the property you want: retries help with isolated failures and are actively harmful during a general outage, and a budget is what tells the difference automatically.
A circuit breaker goes further. Track the failure rate per target, and when it crosses a threshold, stop sending to that target entirely for a cooldown period rather than continuing to probe it with a trickle of doomed requests. After the cooldown, let a small number of requests through, and if they succeed, resume. This protects the target from your pile-on and protects your addresses from accumulating failures against a site that is not answering anyone right now. Both mechanisms are per target rather than global, because one broken site should never stop a job that is collecting from fifty others.
Retries cost money and hide in your data
Two consequences worth stating plainly.
Every retry is a request you pay for. On bandwidth-priced residential proxies a retry storm is a line item, and a job that quietly retries five times against a site that is down all afternoon can move a surprising amount of data for nothing, which is one of the less obvious contributors to the cost picture in cutting proxy bandwidth costs.
And retry rate is a leading indicator that belongs on your dashboard. A rising retry rate on one target is the earliest warning that its defenses changed or your pacing drifted, and it shows up well before success rate collapses, so monitoring the pipeline should track attempts and outcomes rather than only final results. A scraper that silently retries its way to a normal-looking success rate is hiding the problem, not solving it.
Finally, when a request exhausts its attempts, do not throw it away. Push it to a dead-letter queue and retry it much later, in the next run or after a long cooldown, rather than in the current burst. Most items that fail during an incident succeed on their own an hour later, and a queue turns a hard failure into a deferred one.
The bottom line
Retry logic exists to absorb transient failures, not to insist. Classify the failure before acting, retry only what is genuinely transient, and validate response bodies so a soft block is treated as the failure it is. Back off exponentially and always with jitter, honor Retry-After, and cap both delay and attempts. Distinguish rate signals, which call for waiting, from address signals, which call for rotating, and never answer a throttle by cycling addresses to sustain the same pace. Add a retry budget and a per-target circuit breaker so a broad outage cannot turn into a self-inflicted flood. Do these and most of the bans that teams blame on anti-bot escalation simply stop happening, because they were never escalation in the first place.
The other half is having somewhere to fail over to, which is what residential proxies provide: a large pool of real, home-grade IPs so a retired route is replaced by a clean one rather than by the same address trying again. The per-GB pricing is also the reason disciplined retries pay for themselves, since you only pay for the requests you actually make and a storm you never send is bandwidth you never buy.