The architecture that most scraping systems converge on is containers in a cloud provider, scaled by queue depth, stateless and disposable. It is the right design for the compute. It is the worst possible design for the egress, because the addresses those containers get are the first thing a defended site checks.
Cloud provider ranges are published. Anyone can enumerate them, and most anti-bot vendors do. Traffic from them is not blocked because of what it did, it is scored on where it came from before the first request finishes. The fix is not more workers or better headers. It is separating where the code runs from where the traffic exits.
The egress layer is a separate concern
The useful mental model is that your workers are compute and your addresses are identity, and these should scale independently.
Ephemeral workers are good: start fifty, finish the queue, stop. Ephemeral identities are usually bad: every new worker arriving from a new cloud address is a stream of unfamiliar traffic. What you want is a stable set of exit addresses that persists across worker generations, so that the identity your targets see does not churn every time your autoscaler reacts.
ISP proxies fit that role specifically because they are fixed. Each address is dedicated to your account for the life of the plan, registered to a real ISP rather than a cloud block, and does not rotate. Your workers come and go; the addresses do not.
Two auth modes, and the choice is architectural
Every address in a plan uses the same port (1337), with two ways to authenticate. In a cloud deployment this is not a preference, it follows from your networking.
Authorized source IP. You whitelist your infrastructure’s public address in the panel and connect with no credentials:
185.199.108.153:1337This works when your egress is predictable, which in practice means a NAT gateway or equivalent with a stable address in front of your workers. It is the cleaner option when it applies, because there are no credentials in your containers at all.
Username and password. Works from anywhere, no whitelist:
185.199.108.153:1337:USERNAME:PASSWORDThis is what you want when workers are genuinely ephemeral across changing addresses, or spread across regions, or running in an environment where you do not control the outbound address. Serverless functions and multi-region deployments land here.
The trap is picking source-IP auth in an environment where the source address is not actually fixed. It works during testing from one subnet and then fails intermittently in production as the platform allocates elsewhere. If you cannot state your egress address with confidence, use credentials.
Distributing addresses across workers
You have N addresses and a variable number of workers, so something has to assign one to the other.
The approach that holds up is to treat the address list as a shared resource with explicit leases rather than letting each worker pick. A worker takes an address from a coordination layer, holds it for the duration of its job, and returns it. Two properties matter: no two workers use one address simultaneously in a way that doubles its apparent request rate, and a worker crash does not permanently remove an address from circulation.
The naive alternative, hashing worker identity to an index, breaks the moment your worker count changes, which for an autoscaled deployment is constantly. You get collisions under scale-up and idle addresses under scale-down.
Per-address request rate is the real budget here. Bandwidth on ISP plans is unlimited, so the constraint is not gigabytes, it is how much traffic one fixed address can plausibly produce. That number is what determines how many addresses your deployment needs, and it is why sizing by throughput rather than data volume is the correct exercise.
Health checks belong in the pool
Addresses do not fail uniformly. One can start drawing challenges from a specific target while everything else is fine, and a worker that receives that address will produce garbage until something notices.
Track outcomes per address rather than in aggregate. A per-address success rate over a rolling window, per target if you collect from several, gives you the ability to quarantine one address without pulling the whole pool. Aggregate metrics will show you a five percent dip and hide the fact that it is one address failing completely.
Quarantine should be temporary and automatic, with a probe that returns the address when it recovers. The method for establishing what normal looks like in the first place is in testing proxy speed, success rate and location accuracy.
Plan the geography before you buy
One detail that catches cloud teams: on ISP plans you choose the country distribution once, before the addresses are provisioned, and it cannot be changed afterward. Seven countries are available.
This is unlike almost everything else in a cloud deployment, where you are used to changing your mind cheaply. Decide the split against the targets you are actually collecting from, and if you are unsure between two markets, buy the plan you are sure about and add rather than guessing.
When ISP is the wrong tool
Static addresses are not a general replacement for rotation, and cloud infrastructure makes it tempting to pretend otherwise because the operational model is so much simpler.
If your workload is high-volume collection across many targets, the per-address request rate becomes implausible long before the cost does, and the symptom is degraded data rather than errors. That work belongs on rotating residential proxies, where each request can come from a different address by design. The trade-off is set out in ISP proxies vs datacenter and residential.
The workloads where a fixed egress layer is genuinely right: long-lived sessions, anything authenticated, targets that allowlist your addresses, partner APIs that expect a known source, and steady collection where consistency matters more than volume. The mixed architecture is common and correct, with ISP addresses for the persistent work and a rotating gateway for the bulk.
Operational notes
Keep the address list in configuration, not in images. Rebuilding containers to change the pool is avoidable. Read it at startup from your secret store or config service.
Do not log the credentialed URL. It is the single most common way proxy credentials reach a log aggregator, because the natural debug line is the whole connection string.
Bound concurrency per address, not just globally. A global limit spread unevenly across a small pool concentrates load on whichever addresses your assignment logic favours. The relevant mechanics are in rate limiting and request throttling.
Test from inside the deployment. A connectivity check from a laptop proves your credentials work. It proves nothing about whether your task definition passes the environment variables through, which is the actual failure most of the time.
FAQ
Can I use ISP proxies from serverless functions?
Yes, with credential auth. Source-IP whitelisting is impractical there because the outbound address is not yours to fix.
How many addresses does an autoscaled deployment need?
Size by peak concurrent jobs and a conservative per-address request rate, not by worker count. Workers idle; addresses should not be the thing that runs out at peak.
Does the proxy layer become a bottleneck?
ISP addresses run on datacenter infrastructure at gigabit speeds, so throughput per address is rarely the limit. Concurrency policy usually binds first.
Should each service get its own addresses?
If their traffic patterns differ substantially, yes. Mixing a steady low-rate service with a bursty one on the same address means the bursty one determines how both are treated.
The bottom line
Cloud scraping infrastructure gets blocked for a reason that has nothing to do with the quality of the code: the addresses are published as cloud addresses. Putting a fixed ISP egress layer in front of ephemeral workers separates identity from compute, which is the separation the architecture needed anyway.
Lease addresses explicitly, track health per address, size by request rate rather than bandwidth, and decide the geography before you buy, because that choice is permanent. Plans are on the ISP proxies pricing page.