How to Configure Residential Proxies in Docker Containers

Wiring a residential proxy into a container looks trivial until it isn’t. You set HTTP_PROXY, run the image, and requests still leave on the host IP. Or the proxy works but your health checks start failing because internal service calls are now being tunneled through a residential exit in another country. Or, worst of all, your credentials end up baked into an image layer.

None of these are exotic. They’re the four things that reliably bite teams putting proxies in containers: where the config applies, what should bypass it, how secrets get in, and how identity maps to containers. This guide covers each, with working config, for DevOps and platform engineers.

The one distinction that explains most confusion

Docker has two completely separate proxy concepts, and mixing them up is the root of most “why isn’t my proxy working” tickets:

• Build-time / daemon proxy — configured in ~/.docker/config.json or via --build-arg. This governs the Docker daemon (pulling images) and the build process (RUN apt-get install). It has nothing to do with your app’s runtime traffic.
• Runtime proxy — environment variables inside the running container. This is what your application code sees.

If you configured a proxy in ~/.docker/config.json and expected your Python scraper to use it, that’s the bug. Those settings never reach the container’s runtime environment.

And one more layer: even at runtime, HTTP_PROXY is a convention, not enforcement. The kernel does not route traffic through it. Each library chooses whether to honor it:

So the first debugging question is never “is the env var set?” but “does this client read it?”

Runtime proxy: the basic setup

Pass the proxy at run time, not build time, and reference the gateway with targeting encoded in the username:

docker run --rm \
  -e HTTP_PROXY="http://${SHIFTER_USER}-country-us:${SHIFTER_PASS}@p.shifter.io:443" \
  -e HTTPS_PROXY="http://${SHIFTER_USER}-country-us:${SHIFTER_PASS}@p.shifter.io:443" \
  -e NO_PROXY="localhost,127.0.0.1,postgres,redis,.internal,169.254.169.254" \
  my-scraper:latest

Set both upper and lower case (HTTP_PROXY and http_proxy) if you’re unsure about your libraries; conventions differ, and some tools only read one. Note that HTTPS_PROXY still points at an http:// URL, that’s correct: the scheme describes how you talk to the proxy, and HTTPS is tunneled through it with CONNECT.

In Compose, keep the values out of the file itself:

services:
  scraper:
    image: my-scraper:latest
    environment:
      HTTP_PROXY: "http://${SHIFTER_USER}-country-us:${SHIFTER_PASS}@p.shifter.io:443"
      HTTPS_PROXY: "http://${SHIFTER_USER}-country-us:${SHIFTER_PASS}@p.shifter.io:443"
      NO_PROXY: "localhost,127.0.0.1,postgres,redis,.internal"
    depends_on: [postgres, redis]

The variables interpolate from your shell or a .env file that is not committed.

NO_PROXY: the step people skip, and regret

This is the one that causes the confusing failures. Once HTTP_PROXY is set, every honoring client sends everything through the proxy, including calls to your database, your cache, your internal APIs, and cloud metadata endpoints. Consequences: internal traffic leaves your network and comes back (slow, and sometimes broken), health checks fail, and you burn per-GB bandwidth on traffic that should never have left the host.

Always set NO_PROXY to cover:

• localhost, 127.0.0.1, ::1
• Compose/Kubernetes service names (postgres, redis, api.default.svc.cluster.local)
• Internal domains and private ranges (.internal, 10.0.0.0/8)
• Cloud metadata: 169.254.169.254

Two caveats worth knowing: NO_PROXY matching is not consistently implemented across libraries (CIDR support in particular is spotty, and some match suffixes while others need a leading dot), so verify rather than assume. And in Kubernetes, .svc.cluster.local and your pod/service CIDRs belong in NO_PROXY too.

Secrets: don’t bake credentials into the image

Proxy credentials are secrets. The rules:

Never put them in a Dockerfile via ENV or ARG, both persist in image layers and docker history will happily print them. Anyone who can pull the image has your credentials.

Do inject at runtime instead. For Compose, an env file kept out of git; for orchestration, a real secret store:

# Kubernetes: credentials from a Secret, not the manifest
env:
  - name: SHIFTER_USER
    valueFrom:
      secretKeyRef: { name: proxy-creds, key: username }
  - name: SHIFTER_PASS
    valueFrom:
      secretKeyRef: { name: proxy-creds, key: password }

Then build the proxy URL inside the app from those two variables, so the full credential string never appears in a manifest, a log line, or docker inspect. If you need the proxy during build (installing packages), use BuildKit secrets (--mount=type=secret) rather than ARG, so nothing lands in a layer.

Also: scrub proxy URLs from logs. A crash dump that prints the effective config will leak user:pass@host straight into your log aggregator.

Mapping identity to containers

Here’s where container architecture meets proxy architecture. Because targeting lives in the username, each container can carry its own identity just by getting a different env var, no separate endpoints, no IP lists.

Two patterns cover most needs:

One container, one geo. Run per-market workers by varying the country flag:

docker run -d -e HTTP_PROXY="http://${U}-country-us:${P}@p.shifter.io:443" scraper:latest
docker run -d -e HTTP_PROXY="http://${U}-country-de:${P}@p.shifter.io:443" scraper:latest

One container, one sticky session. Give each replica a distinct sid so it holds its own exit IP, useful when a container owns a multi-step flow:

services:
  worker:
    image: scraper:latest
    environment:
      # {{.Task.Slot}} gives each Swarm replica a unique, stable session id
      HTTP_PROXY: "http://${U}-country-us-sid-w{{.Task.Slot}}-ttl-600:${P}@p.shifter.io:443"
    deploy:
      replicas: 4

A caution: a container-level env var is a static identity for that container’s lifetime. If your workload needs rotation per request or per work unit, don’t fake it by restarting containers, set the proxy in application code where you can vary sid per job (the pattern in the load balancing post). Env vars are the right tool for coarse, per-container identity; code is the right tool for fine-grained rotation.

Clients that ignore the environment

Two common cases you’ll hit inside containers:

Node’s fetch/undici doesn’t read proxy env vars. Wire it explicitly:

import { ProxyAgent, setGlobalDispatcher } from "undici";
setGlobalDispatcher(new ProxyAgent(process.env.HTTP_PROXY));

Headless browsers ignore them too, Chromium needs its proxy passed at launch, and credentials handled through the framework’s own mechanism (Playwright covers the specifics, including why inline credentials fail in Chromium). Python’s requests/httpx do honor the env vars, though passing proxies explicitly is clearer (Python guide).

Also worth noting: SOCKS5 through env vars is unreliable across clients in containers. For most container workloads, HTTP proxying is the right default (SOCKS5 tradeoffs).

Verify it from inside the container

Never assume, check the exit IP from within the running container:

docker exec -it my-scraper sh -c \
  'curl -s http://ip-api.com/json | head -c 200'
# Expect a residential IP in the targeted country, not your host IP.

If it returns your host IP, the client isn’t honoring the env vars (see the table above). Add this as a startup assertion in staging so a misconfigured deploy fails loudly instead of quietly scraping from your datacenter IP, and verify NO_PROXY works by confirming an internal service call doesn’t traverse the proxy. Broader measurement methods are in how to test proxy speed, success rate, and location accuracy.

Container gotchas that waste an afternoon

• localhost means the container. A proxy on the host isn’t reachable at 127.0.0.1 from inside; use host.docker.internal (Docker Desktop) or the host’s network address.
• Build-time and runtime proxies are different. Setting one does not set the other.
• Env changes need a recreate. Editing environment in Compose requires up --force-recreate, not a restart.
• DNS resolution location varies. Some clients resolve locally, others let the proxy resolve. If geo-sensitive results look wrong, that’s a suspect.
• Images may lack CA certificates. Slim/alpine bases sometimes need ca-certificates installed for HTTPS through the proxy to validate.
• Per-GB billing is per-container. Ten replicas pulling full pages multiply your bandwidth by ten, cut bandwidth costs applies per replica.

FAQ

Why isn’t my container using the proxy even though HTTP_PROXY is set? Because HTTP_PROXY is a convention, not routing. The library has to honor it. Node’s fetch and headless browsers don’t; curl, Python requests, and Go’s default transport do. Check your client first, then confirm the exit IP from inside the container.

Should I set the proxy in the Dockerfile? No. ENV/ARG values persist in image layers and appear in docker history, leaking credentials to anyone who can pull the image. Inject at runtime via env vars from a secret store, and use BuildKit secrets if you need a proxy during build.

How do I stop internal traffic going through the proxy? Set NO_PROXY with localhost, your service names, internal domains, private ranges, and 169.254.169.254. Otherwise database and health-check traffic tunnels out through a residential exit, which is slow, fragile, and billable.

Can each container have a different IP or country? Yes. Targeting is encoded in the proxy username, so a different env var per container gives each its own country or sticky session, with no extra endpoints. For per-request rotation, set the proxy in application code instead.

Does the proxy work for docker build? Only if you configure the build/daemon proxy separately (~/.docker/config.json or build args). Runtime container env vars don’t affect builds, and build-time credentials shouldn’t be passed as ARG.

The bottom line

Most Docker proxy problems come down to four things. Know where the config applies (build vs runtime, and which clients honor env vars at all), set NO_PROXY so internal traffic stays internal, keep credentials out of image layers and inject them at runtime, and decide deliberately how identity maps to containers (env vars for coarse per-container identity, application code for per-request rotation). Then verify the exit IP from inside the container rather than trusting the config.

Get those right and containerized collection is boring in the best way. The residential gateway helps here because targeting rides in the username: one endpoint, and any container’s geo or session is just a different environment variable. Pool quality still determines how often you’re retrying at all (IP reputation), and the pricing page has the per-GB plans, worth remembering that bandwidth scales with your replica count.