How to Scale Residential Proxy Scraping Workloads with Kubernetes

A scraping workload is close to the ideal Kubernetes tenant: it is embarrassingly parallel, mostly stateless per task, and bursty in a way that begs for autoscaling. Once you have proxies working inside a container, moving to a cluster is the natural next step for anyone running collection at real volume. But the proxy layer changes a few of the defaults you would reach for otherwise: how identity is assigned per pod, what signal you autoscale on, and why adding pods is not a free lunch against the sites you hit.

This is the orchestration layer above the single-container setup. It assumes you already know how to point a scraper at a residential gateway; here we focus on running many of them at once, safely, on Kubernetes.

Throughout, the gateway is Shifter’s: one endpoint, p.shifter.io:443, with targeting encoded in the username (customer-USERNAME-country-us-sid-abc123-ttl-600). Swap host and credentials for another provider; the patterns are identical.

The shape of the workload

Two patterns cover almost everything:

• A Deployment of long-running workers that pull tasks from a queue. This is the default for continuous collection. Pods stay up, drain a work queue, and scale with backlog.
• A Job or CronJob for finite batches: a nightly crawl, a one-off backfill. Kubernetes runs N pods to completion and stops.

Both share the same building blocks below. The queue-driven Deployment is what most teams want, so the manifests here target that, with notes where a Job differs.

Proxy credentials belong in a Secret, never the image

The first rule is boring and non-negotiable: proxy credentials do not go in the image, in a ConfigMap, or hardcoded in a manifest. They go in a Secret, mounted as environment variables (or a file) at runtime.

apiVersion: v1
kind: Secret
metadata:
  name: proxy-credentials
type: Opaque
stringData:
  SHIFTER_USER: "customer-yourname"
  SHIFTER_PASS: "your-gateway-password"

Baking credentials into an image means anyone who can pull the image has your proxy account. A Secret keeps them out of your registry and your git history, and lets you rotate without rebuilding. In production, back this with a real secret manager (External Secrets Operator, Vault, or your cloud’s CSI driver) rather than a plain manifest you might commit by accident.

Give each pod its own identity with the Downward API

Here is the proxy-specific part. If every pod authenticates with the exact same username and no session token, they all draw from the pool the same way, which is usually fine for pure rotation. But the moment you want each worker to hold a coherent, distinct session, one identity per pod rather than one shared blob, you need each pod to vary the sid in its username.

The clean way to do that on Kubernetes is the Downward API: inject the pod’s own name as an environment variable, and derive the session token from it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: scraper
spec:
  replicas: 6
  selector:
    matchLabels: { app: scraper }
  template:
    metadata:
      labels: { app: scraper }
    spec:
      terminationGracePeriodSeconds: 90   # let in-flight requests finish (see below)
      containers:
        - name: scraper
          image: your-registry/scraper:1.4.0
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name        # e.g. scraper-7c9f-abcde
            - name: SHIFTER_USER
              valueFrom:
                secretKeyRef: { name: proxy-credentials, key: SHIFTER_USER }
            - name: SHIFTER_PASS
              valueFrom:
                secretKeyRef: { name: proxy-credentials, key: SHIFTER_PASS }
          resources:
            requests: { cpu: "100m", memory: "192Mi" }
            limits:   { cpu: "500m", memory: "384Mi" }

In the worker, build the proxy username from POD_NAME so each pod is a stable session for its lifetime:

import os, hashlib

user = os.environ["SHIFTER_USER"]
pod  = os.environ.get("POD_NAME", "local")
# Deterministic per-pod session id, stable for the pod's lifetime.
sid  = hashlib.sha1(pod.encode()).hexdigest()[:10]

proxy_user = f"{user}-country-us-sid-{sid}-ttl-600"
proxy = f"http://{proxy_user}:{os.environ['SHIFTER_PASS']}@p.shifter.io:443"
# hand `proxy` to requests / httpx / your client of choice

Now pod scraper-7c9f-abcde is a different session from scraper-7c9f-fghij, deterministically, without any coordination between them. Rotate within a pod by changing the sid per unit of work instead of per pod; the sticky vs rotating distinction applies exactly as it does anywhere else. (For the general worker code that goes inside the container, the Python guide covers client setup and rotation.)

Autoscale on queue depth, not CPU

The default HorizontalPodAutoscaler scales on CPU. For a scraper that is almost entirely I/O-bound, waiting on network round-trips through the proxy, CPU is close to flat regardless of how much work is queued. Scaling on it is nearly useless: you will sit at low CPU with a mountain of backlog and never scale up.

Scale on the backlog itself. The cleanest tool is KEDA, which scales a Deployment on external metrics like the length of a Redis list or the depth of a queue:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: scraper-scaler
spec:
  scaleTargetRef:
    name: scraper
  minReplicaCount: 2
  maxReplicaCount: 40
  triggers:
    - type: redis
      metadata:
        address: redis.default.svc:6379
        listName: scrape:queue
        listLength: "50"     # aim for ~50 queued tasks per pod

This gives you the property you actually want: pods appear when there is work and drain away when the queue empties, so you are not paying for idle workers or starving a backlog. A maxReplicaCount ceiling is not optional, it is your primary guardrail against the next problem.

Concurrency scales with pods, and that is the trap

The single most important thing to internalize: total pressure on a target is per-pod concurrency multiplied by pod count. Each pod might politely cap itself at 5 concurrent requests, but 40 pods is 200 concurrent requests against whatever you are scraping. Autoscaling that felt like a throughput win becomes a distributed hammer that gets your whole pool flagged.

Two defenses, used together:

• Cap per-pod concurrency and keep it modest. A small semaphore per worker.
• Bound total concurrency per target host across the fleet, not just per pod. This is a cluster-level concern that a single pod cannot see, so it belongs in the work queue: shard or rate-limit tasks by target domain so no single site receives more than it tolerates no matter how many pods are running. This is the distributed version of the same logic in proxy load balancing, and it is why a good residential pool is a tool for spreading load across IPs rather than a licence to intensify it.

If you skip this, autoscaling does not make you faster, it makes you blocked. More parallelism past a target’s tolerance buys challenges, not data.

Graceful shutdown: finish or requeue, never drop

Kubernetes kills pods routinely: scale-down, rolling updates, node drains, spot reclaims. When it does, it sends SIGTERM, waits terminationGracePeriodSeconds, then sends SIGKILL. A scraper that ignores SIGTERM loses every in-flight request the instant it is killed, which shows up as mysterious gaps in your data and wasted bandwidth on half-finished fetches.

Handle the signal. On SIGTERM, stop pulling new tasks, let in-flight requests finish within the grace period, and requeue anything you cannot complete in time so another pod picks it up:

import signal

draining = False

def handle_sigterm(signum, frame):
    global draining
    draining = True          # stop pulling new work; finish what is in flight

signal.signal(signal.SIGTERM, handle_sigterm)

while not draining:
    task = queue.pull()
    if task is None:
        continue
    try:
        process(task)        # fetch through the proxy, store result
    except Exception:
        queue.requeue(task)  # at-least-once: let another pod retry

Set terminationGracePeriodSeconds comfortably above your longest expected request, and rely on an at-least-once queue so a killed task is retried rather than lost. This is the pod-level companion to failover in multi-region pipelines: the queue is the durable buffer that makes any single pod disposable.

Health probes that understand the proxy

Liveness and readiness probes decide whether a pod is kept and whether it receives work. For a scraper, tie them to something real. A pod whose proxy egress is broken, wrong exit IP, repeated auth failures, a dead session, should fail readiness so the scheduler stops sending it tasks, and fail liveness if it stays broken so it gets restarted with a fresh identity.

Keep the probe cheap and honest: a lightweight internal check that the last few requests succeeded, not a live call out to the proxy on every probe (that spends bandwidth and can itself time out). The goal is to notice a pod that has quietly stopped exiting through a healthy IP before it burns through a chunk of your queue returning nothing.

Right-size resources: scrapers are I/O-bound

Do not over-provision. A scraping pod spends most of its life blocked on network I/O through the proxy, not burning CPU. Modest requests (a fraction of a core, a couple hundred MB) let the scheduler pack many pods per node, which is exactly what you want for a fleet of light, concurrent workers, the unlimited concurrent connections model in cluster form. Set limits to catch runaways, but keep requests lean so autoscaling can actually place pods. Watch memory more than CPU: a scraper that buffers large responses is far more likely to hit an OOM limit than a CPU one.

Roll out without thundering the target

A rolling update that replaces 40 pods at once creates a burst of fresh sessions all hitting your targets simultaneously, a self-inflicted traffic spike. Use a conservative maxSurge/maxUnavailable so replacement is gradual, and lean on the same queue-and-cap discipline so a deploy does not translate into a request spike. The target should not be able to tell you deployed.

Verify and observe

Two things are worth wiring in from day one. First, an exit-IP check at pod startup that logs the pod’s outbound IP and country, so a misconfigured Secret or a pod stuck on local egress is obvious immediately rather than after a wasted run. Second, per-pod success rate and latency, because in a fleet the failures are rarely uniform: one bad node, one exhausted session, or one target that started challenging you shows up as a single pod’s metrics diverging. Pool quality drives how often you retry at all, so track it, clean IPs with good reputation mean fewer retries per pod and less total load for the same data, and keep an eye on latency and timeout rates as your early warning that a target’s posture changed.

The bottom line

Kubernetes is a strong fit for scraping because the workload is parallel and bursty, and the platform’s autoscaling, self-healing, and rollout machinery map cleanly onto a fleet of workers. The proxy layer adds a short list of rules on top: keep credentials in a Secret, give each pod a distinct session via the Downward API, autoscale on queue depth rather than CPU, remember that total pressure on a target is per-pod concurrency times pod count and cap it fleet-wide, and handle SIGTERM so a killed pod requeues instead of dropping work. Get those right and you can scale collection from six pods to sixty without the target ever feeling it, which is the whole point.

For the pool underneath it, our residential proxies are built to distribute a fleet’s load across many clean IPs, and because they are priced per gigabyte, scaling pods multiplies your throughput without multiplying your proxy bill, only the data you actually move counts. If you are running collection at large scale, that is the combination that keeps it both fast and quiet.