How to Plan Request Volume and Concurrency on Residential Proxies

Most capacity questions arrive backwards. Someone asks how many threads to run or how much bandwidth to buy, when neither is knowable until you have translated the actual requirement, which is usually something like “fifty thousand product records, refreshed daily, ready by 08:00”. That sentence contains everything you need. Here is how to turn it into request volume, concurrency, and a plan size, and where the constraint really binds.

Start from records, not requests

The first conversion is the one people skip, and it is where estimates go wrong by multiples.

One record rarely equals one request. A product record might need a listing page plus a detail page, so two. If the listing paginates and you need every item, add the pagination requests, amortised across the records they yield. If a detail page loads its data from a second call, that is another. And if you render pages in a browser rather than fetching data endpoints, one logical request becomes dozens of asset fetches, which matters enormously for bandwidth even when it does not change the logical count.

So write it out explicitly:

requests_per_record = detail_pages + (listing_pages / records_per_listing) + extra_calls

For fifty thousand records at, say, 1.2 requests each, that is 60,000 requests per run. Then add failure headroom, because your validated success rate is not 100%. At 90% you need roughly 67,000 attempts to land 60,000 successes, and if you are also retrying transient failures the real number is a little higher again. Planning against successes rather than attempts is the second most common estimating error.

Concurrency follows from rate and latency

Now the piece that surprises people. Concurrency is not a number you choose freely; it follows from how fast you need to go and how long each request takes.

If you must complete 67,000 requests in a four-hour window, that is about 4.7 requests per second sustained. Residential requests are slower than direct ones, so assume an average of two seconds per request end to end. The in-flight count you need is simply rate multiplied by latency:

concurrency = requests_per_second x average_latency_seconds
            = 4.7 x 2
            ~ 10 concurrent requests

That relationship is worth internalising, because it explains two things at once. Slower targets need more concurrency for the same throughput, which is why a target that degrades quietly can starve a schedule without any errors appearing. And raising concurrency does not raise throughput if the target is what is slowing you down; it just increases the number of requests waiting.

Work it the other way too. If you cap concurrency at 10 and latency drifts from two seconds to five, your throughput falls from 5 requests per second to 2, and a four-hour job becomes a ten-hour one. Building the schedule with headroom rather than at the limit is what stops a latency drift becoming a missed deadline.

The constraint is the target, not your machine

Here is where planning meets reality. The concurrency your infrastructure can sustain is almost never the binding limit. What binds is what the target tolerates.

Rate limits are enforced per IP and, increasingly, per target in aggregate. A pool lets you spread the per-IP load, but the total arriving at one origin is still visible, so the number to plan against is what that site accepts rather than what your workers can emit. The pacing mechanics are in rate limiting and throttling, and the distribution question that follows, how much spread your volume needs, is worked through in how many proxy IPs you actually need.

Practically this means your plan should carry a per-target concurrency cap and a per-target rate, not one global setting. Fifty targets at modest concurrency each is a completely different proposition from the same total aimed at one site, and only the second gets you blocked.

Bandwidth is the number you actually buy

Since residential proxies bill by data transferred, the plan size follows from bytes rather than from requests or addresses.

monthly_bandwidth = attempts_per_run x runs_per_month x average_bytes_per_response

The variable that dominates is the last one, and it is entirely under your control. A JSON response or a lean HTML page is tens of kilobytes; a fully rendered page with images, fonts, and third-party scripts is several megabytes. At 60,000 daily requests, 50 KB responses come to roughly 90 GB a month, while 2 MB rendered pages come to about 3.6 TB, from the same logical workload. That is the difference between a modest plan and an enterprise one, and it is decided by whether you fetch data or render pages, per when you need a headless browser and cutting bandwidth costs.

Do that optimisation before sizing the plan, because it frequently moves you down a tier. The fuller forecasting method is in estimating monthly bandwidth, and the product-level choices in choosing the right plan.

Scheduling: spread beats burst

Two jobs with identical daily volume can behave completely differently depending on when they run.

Compressing everything into a one-hour window multiplies your instantaneous rate against every target at once, which is precisely the shape that triggers defences. Spreading the same work across the available window lowers per-target rate for free, and it also gives failures room to be retried later in the run rather than piling up at the end.

Freshness requirements set the constraint. If data must be current as of 08:00, you need it collected before then, but that is a deadline rather than an instruction to start at 07:00. Where a schedule allows, spreading across the whole window and prioritising the most volatile sources early is strictly better.

Validate with a pilot before committing

Every number above is a hypothesis until you measure the two variables you assumed: real average latency on your actual targets through residential exits, and real bytes per response after your fetch strategy is settled. Both are easy to measure on a small run and both move the plan materially.

Run a pilot at maybe five percent of intended volume, record validated success rate, latency percentiles, and bytes per request per target, then recompute. Expect the answer to differ from your estimate in both directions. The measurement method is in testing speed, success rate and location accuracy, and the ongoing versions of those same numbers are the metrics in proxy KPIs.

A worked summary

Requirement: 50,000 records daily, ready by 08:00, collection window 04:00 to 08:00.

Requests per record 1.2, giving 60,000 successes. At 90% validated success, about 67,000 attempts. Over four hours that is 4.7 per second. At two seconds average latency, roughly 10 concurrent in flight, which you then split across targets rather than pointing at one. At 80 KB average per response, about 5.4 GB per run and roughly 160 GB per month. Add headroom on both concurrency and bandwidth, because latency drifts and success rates dip, and re-derive after the pilot.

The bottom line

Plan backwards: records to requests, requests to attempts after failure headroom, attempts over the window to a rate, rate times latency to concurrency, and attempts times response size to bandwidth. Cap concurrency and rate per target rather than globally, because the binding constraint is what each site tolerates and not what your infrastructure can produce. Optimise what you fetch before you size the plan, since rendering versus fetching data can change the bandwidth answer by two orders of magnitude. Spread the work across the available window rather than bursting. Then run a pilot, because latency and bytes per response are the two assumptions worth replacing with measurements before you commit to a plan.

The capacity itself comes from residential proxies, where concurrency is not the constraint the plan is sized on, with per-GB pricing so the bandwidth figure you derive is the number you are actually buying.