Rate Limiting and Request Throttling with Residential Proxies

There is a tempting piece of logic that catches out a lot of teams: rate limits are enforced per IP, a residential pool gives you many IPs, therefore rate limits no longer apply. The first two clauses are true and the conclusion is wrong, and the gap between them is where a surprising number of blocked scrapers live.

A pool changes where the limit binds, not whether one exists. You still have to pace, just along different axes than a single-IP setup, and the axis people forget is the one that gets them caught. Here is how throttling actually works when your requests are spread across a rotating pool.

Three limits, not one

When you scrape through a pool, at least three separate ceilings apply at once, and your throughput is set by whichever you hit first.

Per IP, per target. This is the classic one. Any single address can only send so many requests to a given site before that site throttles or blocks it. Rotation is what keeps you under this, which is the whole point of distributing load across a pool as described in load balancing.

Per target, in aggregate. This is the one that surprises people. A site does not only count per address; it also sees its own total inbound load and can recognise a coordinated pattern across many addresses, particularly if the requests share timing, paths, or behaviour. Your fifty IPs sending two requests each per second is a hundred requests per second arriving at one origin, and no amount of address rotation makes that look like ordinary organic traffic. Anti-bot systems increasingly reason at this level.

Your own capacity. Concurrency you can actually sustain: open connections, worker threads, memory, and the fact that residential latency is higher than a direct connection, so a fixed thread count yields fewer requests per second than you might expect.

The practical consequence is that your throttle needs to be expressed per target rather than globally. A single global rate limit either starves you on the fifty sites that could take more, or hammers the one that cannot.

Start from the target, not from your ambitions

Before writing any throttling code, find out what the target actually tolerates, because guessing produces either a needlessly slow job or a blocked one.

Read the signals the site gives you. A 429 is an explicit statement that you are going too fast. A Retry-After header is the site telling you exactly how long to wait, and honouring it is both correct and much cheaper than discovering the answer by trial. Some APIs publish limits in documentation or return remaining-quota headers. And if the site has a robots.txt with a crawl-delay directive, that is a stated preference worth respecting.

Where nothing is published, calibrate empirically: start conservatively, increase gradually, and watch validated success rate rather than status codes, since a site under strain often degrades before it refuses, and a challenge page with a 200 status looks like success to a naive counter, as covered in detecting blocked or fake content. The point at which success rate starts sliding is your real ceiling, and you want to operate below it rather than at it.

Implementing the throttle

A token bucket per target is the standard tool and is simple enough to write from scratch. Tokens refill at your chosen rate, each request consumes one, and a request waits when the bucket is empty. That gives you a steady average with a controlled burst allowance, which matches how real traffic behaves better than a rigid one-request-per-N-seconds delay.

import time, threading

class TargetLimiter:
    """Token bucket, one instance per target host."""
    def __init__(self, rate_per_sec, burst=5):
        self.rate, self.capacity = rate_per_sec, burst
        self.tokens, self.updated = burst, time.monotonic()
        self.lock = threading.Lock()

    def acquire(self):
        while True:
            with self.lock:
                now = time.monotonic()
                self.tokens = min(self.capacity,
                                  self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
                wait = (1 - self.tokens) / self.rate
            time.sleep(wait)               # sleep outside the lock

LIMITS = {                                  # per target, tuned per target
    "shop.example.com":  TargetLimiter(2.0),
    "search.example.com": TargetLimiter(0.5),
}

def fetch(url, host, proxies):
    LIMITS[host].acquire()
    return requests.get(url, proxies=proxies, timeout=20)

Two refinements matter in production. Add jitter so requests do not land on exact intervals, since perfectly regular spacing is itself a machine signature and it also synchronises your workers into waves. And make the limiter shared across workers rather than per process, otherwise ten processes each politely doing two requests per second are collectively doing twenty. In a distributed setup that means a shared counter in Redis or similar, which is the same problem scaling scraping across Kubernetes has to solve.

Concurrency is the other half

Rate and concurrency are different dials and both need limits. Rate controls how often you start requests; concurrency controls how many are in flight simultaneously. A job with a modest rate limit but unbounded concurrency will still open hundreds of simultaneous connections to one origin the moment that origin slows down, because slow responses cause requests to pile up.

Cap concurrency per target with a semaphore, and size it against what the target tolerates rather than what your machine can open. On the proxy side, concurrent connections are generally not your constraint, which makes it easy to forget that the target’s tolerance is. The distribution question of how many addresses that implies is worked through in how many proxy IPs you actually need.

Adaptive throttling beats a fixed number

A static rate is a guess that becomes wrong. Sites change their tolerance, get slower under load, or tighten limits during peak hours. The more robust pattern is to adjust based on what you observe, in the style of additive increase and multiplicative decrease: creep the rate up slowly while things are healthy, and cut it sharply at the first sign of strain.

Trigger the decrease on a composite signal rather than one status code: a 429, a rising share of challenge pages, climbing latency, or a fall in validated success rate. Then recover gradually rather than jumping straight back to the old rate, since an immediate return to full speed after a block is a recognisable pattern in itself.

class AdaptiveRate:
    def __init__(self, start=2.0, floor=0.2, ceiling=10.0):
        self.rate, self.floor, self.ceiling = start, floor, ceiling

    def ok(self):                       # healthy response
        self.rate = min(self.ceiling, self.rate * 1.02)   # creep up

    def strained(self):                 # 429, challenge, timeout, latency spike
        self.rate = max(self.floor, self.rate * 0.5)      # back off hard

Where throttling meets rotation and retries

Three interactions are worth being explicit about, because getting them wrong undoes the throttle.

Do not answer a rate signal by rotating. A 429 means slow down. Swapping to a fresh address so you can keep the same pace is the behaviour that turns one throttled address into a burned pool, and it is precisely what makes traffic look evasive rather than merely enthusiastic. Wait when the signal is about rate; rotate when the signal is about the address.

Retries must respect the limiter. A retry is another request and must acquire a token like any other, or your error path quietly bypasses the pacing you built. Retry storms are the most common way a throttled job becomes a blocked one, which is the subject of retry and backoff.

Sticky sessions concentrate load. Holding one address for a multi-step flow means that address carries the whole sequence, so per-IP pacing matters more inside a sticky session than in rotating work where the load naturally spreads.

Politeness is self-interest

It is easy to read all of this as a compliance chore, but the incentives point the same way. Pacing preserves the reputation of the addresses you depend on, keeps your success rate high so you are not paying bandwidth for challenge pages, and avoids the escalation cycle where aggressive collection provokes tighter defences that make the target harder for everyone, yourself included next quarter. Respecting a site’s stated limits, its robots directives, and its terms is both the right thing and the cheaper thing.

The bottom line

A pool does not remove rate limits, it moves them. Throttle per target rather than globally, because a single number is wrong for every site you touch. Find each target’s real tolerance from its own signals, honour Retry-After and 429 as instructions rather than obstacles, and calibrate against validated success rate instead of status codes. Implement with a shared token bucket plus jitter, cap concurrency separately from rate, and make the limit adaptive so it degrades quickly and recovers slowly. Then keep the interactions straight: never answer a pacing signal by rotating addresses, always route retries through the limiter, and pace harder inside sticky sessions. Done well, throttling is what lets a pool run at its actual capacity instead of burning through it.

The pool itself is what gives you room to work: residential proxies spread a properly paced job across many real home-grade addresses with country and city targeting, and per-GB pricing means a job that paces well and fetches only what it needs costs less than one that hammers and retries.