Most scraping codebases grow a proxy manager whether anyone designs one or not. It starts as a helper that builds a proxy URL, then someone adds a retry, then a special case for a target that needs sticky sessions, then a counter to stop hammering an address that keeps failing. Eighteen months later that logic is spread across four modules and nobody can say what happens when a request fails twice on the same session.
It is worth building deliberately instead, because the decisions involved are genuinely shared: which exit does this request use, is that exit healthy, what happens when it fails, and how do we pace it. Here is a design for that component and the state it needs to hold.
What the manager is responsible for
Draw the boundary tightly. A proxy manager decides how a request leaves your infrastructure and what to do when that fails. It does not parse pages, it does not know what a product listing looks like, and it does not decide which URLs to visit. Keeping it ignorant of your business logic is what lets every job share it.
That leaves four responsibilities: selecting an exit for each request according to a policy, tracking the health of what it hands out, applying failure handling when something goes wrong, and enforcing pacing so callers cannot individually overwhelm a target. Everything else belongs elsewhere.
Selection: the policy is per job, not global
The first thing the manager needs is a notion of the kind of exit a caller wants, rather than a single global mode. In practice there are three.
Rotating, where each request gets a fresh exit. This is the default for bulk collection and is expressed by omitting a session identifier, which is how the rotation model is designed to be used.
Sticky, where a caller holds one exit for a sequence, and the manager must hand back the same session identifier for the life of that sequence, then release it. The trade-offs are in sticky versus rotating.
Geo-pinned, where the exit must be in a specific country or city, which for multi-market work is orthogonal to the first two: a job can be rotating-in-Germany or sticky-in-Chicago.
Model those as a request-scoped lease rather than a global setting, so a caller asks for what it needs and the manager satisfies it:
from dataclasses import dataclass
from typing import Optional
import itertools, time
@dataclass
class Lease:
country: str
session: Optional[str] = None # None means rotate per request
ttl: Optional[int] = None # only meaningful with a session
def username(self, customer="USERNAME"):
parts = [f"customer-{customer}", f"country-{self.country}"]
if self.session:
parts.append(f"sid-{self.session}")
if self.ttl:
parts.append(f"ttl-{self.ttl}")
return "-".join(parts)
def proxies(self, password="PASSWORD", host="p.shifter.io:443"):
url = f"http://{self.username()}:{password}@{host}"
return {"http": url, "https": url}The manager’s job is to produce a Lease, hand it to the caller, and observe what happens to it.
Health: track sessions, not addresses
Here is where most home-grown managers go wrong. On a pooled gateway you do not hold a list of IP addresses to mark good or bad, because you never chose them and you do not get to keep them. What you can track is the health of a session, and the aggregate health of a route, meaning a combination of country and target.
So maintain two things. Per active sticky session, a small record of consecutive failures and challenge responses, so a session that has clearly gone bad can be retired and replaced with a fresh identifier. Per route, a rolling success rate, which tells you whether a whole country-target combination has degraded rather than one unlucky session.
A health check in this world is not a periodic ping. Pinging an exit tells you it can reach a test endpoint, which is not the question; the question is whether it can reach your target and get real content. So use passive health checking: every real request is the health check, and its validated outcome updates the record. Active probing is worth it only for a small, cheap canary against each target, which is useful to distinguish “this target is down for everyone” from “our routes to it have degraded”.
Crucially, judge health on validated responses rather than status codes. A challenge page with a 200 is a failure for health purposes, and a manager that counts it as success will keep reusing a session the target has already decided about. That is the same validation discipline as in detecting blocked or fake content.
class RouteHealth:
def __init__(self, window=50):
self.window, self.results = window, []
def record(self, ok: bool):
self.results.append(ok)
if len(self.results) > self.window:
self.results.pop(0)
@property
def success_rate(self):
return sum(self.results) / len(self.results) if self.results else 1.0
@property
def degraded(self):
return len(self.results) >= 10 and self.success_rate < 0.7Failure handling: classify, then act
The manager owns the decision of what a failure means, which is what stops that logic being reinvented per job. Three classes cover it.
Rate signals mean slow down: a 429 or a Retry-After. The correct response is to wait, and specifically not to swap to a fresh exit so the same pace can continue, since that is what turns a throttled route into a burned one.
Identity signals mean this exit is finished for this target: a block page, a persistent 403, or repeated challenges on one session. The correct response is to retire the session, take a fresh one, and continue, which is the failover pattern.
Terminal errors mean stop: a 404, a malformed URL, or a 407, which is an authentication problem that will be equally wrong on every retry and should fail loudly rather than loop, as covered in fixing 407 and credential errors.
Everything else is transient and gets exponential backoff with jitter, bounded by an attempt cap, per retry and backoff. The important architectural point is that the retry lives in the manager, so every caller inherits the same behaviour, and the manager can enforce a retry budget across callers rather than each job retrying independently into the same struggling target.
Add a circuit breaker per route while you are there. When a route’s health drops below a threshold, stop sending to it for a cooldown, then allow a trickle to test recovery. That protects the target from your pile-on and protects your account from accumulating failures against a site that is not answering anyone.
Pacing belongs here too
Since the manager sees every outbound request, it is the natural place to enforce per-target rate limits and concurrency caps, which is the mechanism described in rate limiting and request throttling. Putting it here has a specific benefit: retries automatically respect the limiter, because they go through the same path as any other request. A retry that bypasses pacing is how a struggling target becomes a blocked one.
Putting it together
The whole surface is small, which is the point:
class ProxyManager:
def __init__(self, limiter_factory, health_factory):
self.limiters = {} # host -> TargetLimiter
self.health = {} # (country, host) -> RouteHealth
self.limiter_factory, self.health_factory = limiter_factory, health_factory
def get(self, url, host, country, session=None, attempts=4):
route = self.health.setdefault((country, host), self.health_factory())
limiter = self.limiters.setdefault(host, self.limiter_factory(host))
sid = session
for attempt in range(attempts):
if route.degraded:
raise RouteUnavailable(country, host) # circuit open
limiter.acquire() # pacing, retries included
lease = Lease(country=country, session=sid, ttl=600 if sid else None)
outcome = self._send(url, lease) # returns (klass, response)
route.record(outcome.klass == "ok")
if outcome.klass == "ok":
return outcome.response
if outcome.klass == "terminal":
raise TerminalError(url, outcome.response)
if outcome.klass == "identity" and sid:
sid = new_session_id() # retire, do not reuse
if outcome.klass == "rate":
time.sleep(outcome.retry_after or backoff(attempt))
else:
time.sleep(backoff(attempt)) # transient
raise Exhausted(url)Two design notes. The manager returns responses and raises typed errors rather than returning None, so callers cannot silently treat a failure as empty data. And it never swallows a terminal error, because an authentication or configuration problem should stop a job rather than be retried into invisibility.
Operational concerns
Make it observable. The manager sees every request, so it is where per-route success rate, retry ratio, and bytes per request should be emitted, which is exactly the data the metrics in proxy KPIs are built from and what pipeline monitoring consumes.
Share state across processes. A per-process manager multiplies your effective rate by the number of workers and fragments health tracking. In a distributed deployment, limiters, breakers, and route health belong in shared storage, which is the same consideration as scaling with Kubernetes.
Keep credentials out of it. The manager composes usernames; it should read the password from a secret store rather than holding it in configuration, and rotation should be a deployment step rather than a code change.
Do not over-abstract. Resist a plugin architecture for hypothetical providers. One gateway, expressed cleanly, is easier to reason about than an abstraction layer over vendors you do not use.
The bottom line
A proxy manager is the component that decides how a request leaves your system and what happens when it fails, and centralising those decisions is what stops the same logic being reimplemented, inconsistently, across every job. Give callers a request-scoped lease so rotation, stickiness, and geography are per job rather than global. Track health at the session and route level rather than pretending to manage individual addresses, and judge it on validated responses so a challenge page counts as the failure it is. Own failure classification in one place: wait on rate signals, retire the session on identity signals, fail loudly on terminal errors, and back off with jitter on everything else, with a circuit breaker per route. Put pacing in the same path so retries cannot bypass it. Then emit metrics from it, because it is the only component that sees everything.
Underneath it sits the residential proxy network: one gateway where country, city, session and TTL are all expressed in the username, which is what makes a manager like this a small amount of code rather than an integration project, with per-GB pricing so the efficiency it buys you shows up directly on the invoice.