How to Cut Proxy Bandwidth Costs When Scraping at Scale

Residential proxies are billed by the gigabyte, so on a per-GB plan your bandwidth is your bill. The catch is that a naive scraper wastes most of the bytes it pays for, downloading images, fonts, videos, and tracking scripts it never looks at, re-fetching pages that haven’t changed, and rendering full browsers when a plain HTTP request would do. The good news: almost all of that waste is easy to cut, and a well-tuned scraper routinely uses less than half the bandwidth of an untuned one for the same data.

This is a practical guide to doing exactly that. Nine tactics, ordered roughly by impact, for collecting the same data through your residential proxies while paying for a fraction of the bytes. It’s the cost-efficiency companion to the per-GB shift covered in bandwidth-priced residential proxies.

1. Don’t render a browser unless you have to

This is the single biggest lever. A headless browser downloads everything the page references: images, CSS, web fonts, video, analytics, ad scripts. For most scraping, you need none of it. If the data you want is in the initial HTML or reachable via an API (next section), skip the browser entirely and use a plain HTTP client, requests, httpx, or Scrapy. A single HTTP GET can be a tiny fraction of the bytes a full browser render pulls.

Reserve the browser for pages that genuinely require JavaScript execution to produce the data. When in doubt, fetch the raw HTML first and check whether your target values are already there.

2. Hit the JSON API, not the HTML

Many sites render their content from a backend JSON endpoint that the page calls after load. Fetching that endpoint directly, instead of the full HTML page and its assets, is often a fraction of the bytes and gives you clean, structured data with no parsing. Open your browser’s network tab, find the request that returns the data (usually an XHR/fetch returning JSON), and call it directly through the proxy. One lean JSON response can replace a heavy page plus a browser render.

3. Block heavy resource types when you must use a browser

When a page really does need rendering, don’t let the browser download what you won’t use. Both Playwright and Puppeteer let you intercept requests and abort images, media, fonts, and often stylesheets before they hit the proxy, which typically cuts bandwidth by well over half. In Playwright:

await context.route("**/*", (route) => {
  const type = route.request().resourceType();
  if (["image", "media", "font"].includes(type)) {
    return route.abort();          // never downloaded, never billed
  }
  return route.continue();
});

Keep document, script, xhr, and fetch; drop the rest. The same pattern and the Chromium specifics are in how to use residential proxies with Playwright.

4. Keep compression on

Text compresses roughly 5 to 10x, so a gzip- or brotli-encoded HTML response is a fraction of the raw size. Most HTTP clients send Accept-Encoding: gzip, deflate, br and decompress transparently by default, but verify you haven’t disabled it, and never strip the header. If you’re setting headers manually, include it:

headers = {"Accept-Encoding": "gzip, br", "User-Agent": "..."}

The bytes that cross the proxy are the compressed ones, so this is a direct, free reduction in what you’re billed.

5. Use conditional requests so unchanged pages cost almost nothing

For any recurring crawl, don’t blindly re-download pages. HTTP conditional requests let the server tell you “nothing changed” in a tiny 304 response instead of resending the whole body. Store the ETag or Last-Modified from each fetch and send it back:

# On refetch, send the stored validator
headers = {"If-None-Match": stored_etag}      # or If-Modified-Since
r = client.get(url, headers=headers)
if r.status_code == 304:
    pass   # unchanged: a few bytes instead of the full page

A 304 costs almost nothing compared to a full page, and on a monitoring job where most pages rarely change, this alone can slash bandwidth.

6. Don’t re-fetch what hasn’t changed

Related, but broader than HTTP validators: track what you’ve already collected and skip it. Use sitemap lastmod timestamps to find changed URLs, keep a content hash per page and skip re-processing identical content, and dedupe your crawl frontier so you don’t fetch the same URL twice in a run. The cheapest gigabyte is the one you never download.

7. Fetch only the bytes you need

If you only need a file’s headers or first chunk (a metadata check, a content-type sniff), don’t download the whole thing. A HEAD request returns headers with no body, and a Range request pulls just the bytes you specify where the server supports it:

client.head(url)                                   # headers only, no body
client.get(url, headers={"Range": "bytes=0-2047"}) # first 2 KB only

This matters most when your crawl touches large media, PDFs, or downloads where you only need a small part or just metadata.

8. Make retries and geo-targeting efficient

Every failed attempt still spends bandwidth, so wasteful retries quietly inflate your bill. Back off and cap retries rather than hammering a failing URL, and avoid over-tight geo filters (country + city + ASN all at once) that cause requests to fail and retry when no matching IP is free, loosen a filter instead. Set sane timeouts so a stalled response doesn’t drag, and match your session strategy to the job so you’re not re-establishing flows unnecessarily. Efficient requests are cheap requests.

9. Measure bytes per record

You can’t cut what you don’t measure. Track bandwidth per job and, better, bytes per successful record extracted, and the waste reveals itself: a page costing megabytes per row is usually pulling assets you could block or rendering when you could have hit an API. Log it, watch the trend, and optimize the worst offenders first.

A note on doing it responsibly

Cutting bandwidth is not just cheaper for you, it’s lighter on the sites you collect from: fewer and smaller requests mean less load on their servers. That aligns cost efficiency with good citizenship. Keep collecting public data, honor each site’s terms and rate limits, and let efficiency reduce your footprint. Our acceptable use policy is the source of truth for what’s allowed on Shifter.

FAQ

What uses the most bandwidth when scraping through proxies? Full browser rendering, by a wide margin, because it downloads every image, font, video, and script on the page. Cutting rendering (or blocking heavy resource types when you must render) is the highest-impact change you can make.

Does blocking images and fonts break scraping? Usually not, you rarely need them for data extraction. Test that the site doesn’t lazy-load target content behind images; for the large majority of jobs, blocking image, media, and font requests is safe and cuts bandwidth substantially.

Do residential proxies charge for failed or blocked requests? The bytes that cross the proxy are billable, so failed attempts and retries still cost bandwidth. That’s why efficient retries, sane timeouts, and avoiding over-tight geo filters (which cause failures) matter for your bill.

Is hitting a site’s JSON API allowed? Calling a public endpoint the page itself uses is common practice and far more efficient, but the site’s terms and rate limits still apply. Collect responsibly and get legal advice for anything uncertain.

How much can I realistically save? It varies by target, but skipping unnecessary rendering, blocking heavy resources, and not re-fetching unchanged pages commonly cut total bandwidth by more than half for the same extracted data.

The bottom line

On a per-GB plan, bandwidth efficiency is cost efficiency, and most scrapers leave a lot on the table: rendering when they could fetch, downloading assets they never read, and re-pulling pages that never changed. Work down this list, skip the browser when you can, hit the API, block heavy resources, keep compression on, use conditional requests, don’t re-fetch unchanged pages, fetch only what you need, make retries efficient, and measure, and the same data comes back for a fraction of the bytes.

That efficiency compounds with pool quality: clean IPs mean fewer blocks and retries, which is fewer wasted bytes, so it’s worth choosing the right proxies for scraping alongside tuning your client. See the per-GB plans on the pricing page, then point a well-tuned scraper at the residential gateway and watch the bytes drop.