How to Use Residential Proxies with Python (requests, httpx, Scrapy)

A hands-on guide to residential proxies in Python: auth, rotating vs sticky IPs, geo-targeting, retries, with examples for requests, httpx, and Scrapy.

Wiring a residential proxy into a Python script is a five-minute job once you’ve seen it done. The friction is never the code, it’s the details nobody writes down: how the credentials encode targeting, when to rotate versus pin an IP, how to handle the proxy-specific errors, and the small library differences between requests, httpx, and Scrapy.

This is the guide I wish I’d had. Copy-paste examples that actually run, for the three libraries you’re most likely using, plus the parts that turn a working snippet into a scraper that survives production.

Everything below uses the Shifter residential gateway: one endpoint, p.shifter.io:443, with all targeting encoded in the username. If you’re on a different provider the shape is the same; swap the host and credential format.

The one thing you have to understand first

On a residential gateway, the proxy username carries your auth and your targeting. You don’t change endpoints to switch country or session, you change the username string. A username looks like this:

customer-USERNAME-country-us-sid-abc123-ttl-600

Read it left to right: account id, then flags. country-us targets the US. sid-abc123 pins a sticky session. ttl-600 holds that IP for 600 seconds. Drop sid/ttl and every request rotates to a new IP. The password is constant. That’s the whole mental model, the rest is just plugging this string into each library’s proxy slot.

Keep credentials in environment variables, never hardcoded:

import os
USER = os.environ["SHIFTER_USER"]   # your account username
PASS = os.environ["SHIFTER_PASS"]
GATEWAY = "p.shifter.io:443"

requests: the 6-line version

requests takes a proxies dict. Both http and https keys point at the same http:// proxy URL, that’s correct, requests tunnels HTTPS through it via CONNECT.

import os, requests

USER = os.environ["SHIFTER_USER"]
PASS = os.environ["SHIFTER_PASS"]
GATEWAY = "p.shifter.io:443"

def proxy(country="us"):
    url = f"http://{USER}-country-{country}:{PASS}@{GATEWAY}"
    return {"http": url, "https": url}

r = requests.get("https://api.ipify.org?format=json", proxies=proxy("us"), timeout=30)
print(r.json())   # {'ip': '<a US residential IP>'}

Run it twice and you’ll get two different IPs, because there’s no sid, every request rotates. That’s the default, and for most scraping it’s exactly what you want.

Rotating vs sticky, in code

This is the distinction that trips people up, so here it is concretely. (For the conceptual version, see sticky vs rotating residential proxies.)

Rotating (new IP every request) is the default, just omit sid:

for _ in range(3):
    r = requests.get("https://api.ipify.org", proxies=proxy("us"), timeout=30)
    print(r.text)   # three different IPs

Sticky (same IP across requests) needs a session id and a TTL in the username. Use it when a flow spans multiple requests that must look like one user, a login, then the pages behind it:

def sticky_proxy(country="us", session="s1", ttl=600):
    url = f"http://{USER}-country-{country}-sid-{session}-ttl-{ttl}:{PASS}@{GATEWAY}"
    return {"http": url, "https": url}

s = requests.Session()
p = sticky_proxy(session="checkout-42", ttl=600)
for path in ("/login", "/cart", "/checkout"):
    r = s.get(f"https://shop.example{path}", proxies=p, timeout=30)
    # all three requests share one IP for up to 600s

Pick the session id yourself, any unique string per logical session. The same string returns the same IP until the TTL expires, then it rotates while keeping your other flags.

Geo-targeting

Because targeting lives in the username, geo is just another flag. Country is the common one; state, city, and ASN work the same way:

def geo_proxy(country, city=None):
    parts = [USER, "country", country]
    if city:
        parts += ["city", city.lower().replace(" ", "_")]
    url = f"http://{'-'.join(parts)}:{PASS}@{GATEWAY}"
    return {"http": url, "https": url}

requests.get("https://example.com", proxies=geo_proxy("de"))              # Germany
requests.get("https://example.com", proxies=geo_proxy("us", "new york"))  # New York City

City names use underscores for spaces (new_york). Combine flags freely; order doesn’t matter to the gateway.

httpx: same idea, async-ready

httpx is the modern choice when you want async or HTTP/2. The proxy goes on the client. Note the parameter is proxy= (singular) in current httpx; older versions used proxies=.

import os, httpx

USER = os.environ["SHIFTER_USER"]
PASS = os.environ["SHIFTER_PASS"]
GATEWAY = "p.shifter.io:443"

def proxy_url(country="us"):
    return f"http://{USER}-country-{country}:{PASS}@{GATEWAY}"

# Sync
with httpx.Client(proxy=proxy_url("us"), timeout=30) as client:
    print(client.get("https://api.ipify.org").text)

The async version is where httpx earns its keep, fan out many requests concurrently, each through a fresh rotating IP:

import asyncio, httpx

async def fetch(client, url):
    r = await client.get(url, timeout=30)
    return r.status_code, r.text[:80]

async def main(urls):
    async with httpx.AsyncClient(proxy=proxy_url("us")) as client:
        return await asyncio.gather(*(fetch(client, u) for u in urls))

urls = ["https://api.ipify.org"] * 10
print(asyncio.run(main(urls)))   # 10 concurrent requests, rotating IPs

That’s ten concurrent residential requests in a dozen lines. Mind your concurrency, more isn’t always faster, and hammering one target from a burst of IPs can still trip behavioral detection.

Scrapy: a proxy middleware

Scrapy is the heavyweight for large crawls. The clean way to attach proxies is to set request.meta["proxy"] per request, Scrapy’s built-in HttpProxyMiddleware reads it and handles the Proxy-Authorization header from credentials in the URL.

A tiny middleware that rotates geo and gives each request a fresh IP:

import os

class ResidentialProxyMiddleware:
    def __init__(self):
        self.user = os.environ["SHIFTER_USER"]
        self.password = os.environ["SHIFTER_PASS"]
        self.gateway = "p.shifter.io:443"

    def process_request(self, request, spider):
        country = request.meta.get("country", "us")
        request.meta["proxy"] = (
            f"http://{self.user}-country-{country}:"
            f"{self.password}@{self.gateway}"
        )

Enable it in settings.py (it must run before the default proxy middleware):

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ResidentialProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 400,
}

Now any spider routes through residential IPs, and you can target per request with Request(url, meta={"country": "gb"}). For a sticky session, build the username with a sid/ttl exactly as in the requests example and set it on meta["proxy"].

Handling the errors that actually happen

A scraper that ignores proxy errors works in the demo and dies overnight. Three you’ll meet:

407 Proxy Authentication Required, wrong username/password, or an unrecognized flag (a typo in country or asn). Fix the credential string; retrying won’t help.
429 Too Many Requests, the target is rate-limiting you. Back off, slow down, and rotate IPs (which the gateway already does per request when you’re not sticky).
502 Bad Gateway, no IP currently matches your filter (usually over-tight geo like country+city+asn). Loosen a flag and retry.

A minimal retry wrapper that backs off and gives up cleanly:

import time, requests

def get_with_retry(url, proxies, tries=4):
    for attempt in range(tries):
        try:
            r = requests.get(url, proxies=proxies, timeout=30)
            if r.status_code in (429, 502, 503):
                raise requests.exceptions.RequestException(f"status {r.status_code}")
            r.raise_for_status()
            return r
        except requests.exceptions.RequestException as e:
            if attempt == tries - 1:
                raise
            sleep = 2 ** attempt           # 1s, 2s, 4s
            print(f"retry {attempt+1}: {e}; sleeping {sleep}s")
            time.sleep(sleep)

Exponential backoff plus per-request rotation handles the large majority of transient failures. If you’re getting blocked consistently rather than intermittently, the problem isn’t retries, it’s IP quality or request behavior, covered in how to avoid getting blocked when scraping.

Verifying it actually works

Before you trust a config, confirm two things: the IP changes (rotation) and it’s in the right country (geo). One quick check:

import requests

r = requests.get("http://ip-api.com/json", proxies=proxy("de"), timeout=30)
data = r.json()
print(data["query"], data["countryCode"])   # expect a DE IP

Call it a few times without a sid and you should see different IPs, all DE. If the country is wrong, check your flag spelling. If the IP never changes, you accidentally left a sid in the username.

A note on SOCKS5

Everything above uses HTTP proxies, the right default for web scraping. If your workload needs SOCKS5 (non-HTTP traffic, or a tool that expects it), the same gateway speaks it, change the scheme to socks5h:// and install requests[socks] (or use httpx’s SOCKS extra). The tradeoffs are in HTTP vs SOCKS5 proxies; for plain scraping, stick with HTTP.

FAQ

Do I need a different endpoint for each country? No. One endpoint, p.shifter.io:443, for everything. Country, city, and session all change via the username string, not the host. That’s the core of the gateway model.

Why are both http and https keys set to an http:// URL in requests? Because requests sends HTTPS through an HTTP proxy via a CONNECT tunnel. The proxy URL scheme describes how you talk to the proxy (HTTP), not the target. This is correct and standard, don’t set the https key to https://.

How do I rotate IPs on every request? Omit sid from the username. With no session id, the gateway hands you a fresh IP per request automatically. You don’t manage a proxy list, the pool rotation is server-side.

How do I keep the same IP across a multi-step flow? Add sid-<your-id>-ttl-<seconds> to the username and reuse it for each request in the flow. Same id, same IP, until the TTL expires.

My requests are slow. Is the proxy the problem? Usually not. Residential IPs add some latency versus a direct connection, but slowness at scale is more often too-high concurrency, no connection reuse (use a Session/Client), or a slow target. Profile before blaming the proxy.

Does this work with aiohttp, urllib3, or pycurl too? Yes. Any client that accepts an authenticated http://user:pass@host:port proxy URL works, the credential-encoded targeting is identical. requests, httpx, and Scrapy are just the three most common.

Wrapping up

The pattern is the same in every library: build the http://USER-flags:PASS@p.shifter.io:443 URL, drop it in the proxy slot, omit sid to rotate or add it to stick. Geo is a flag, errors are a small retry loop, and concurrency is whatever your client already does.

Start from the snippets above, point them at the residential gateway, and you’ve got production-shaped proxy code in an afternoon. Plans and per-GB rates are on the pricing page, and the full flag reference lives in the gateway docs.