How to Rotate Residential Proxies in Scrapy with Custom Middleware

Scrapy is the framework you reach for when a scrape outgrows a script: it handles scheduling, concurrency, retries, and pipelines out of the box. Adding a residential proxy is straightforward, but Scrapy does it differently from a plain HTTP client. The proxy is a per-request setting handled by a downloader middleware, and that architecture has one specific trap around authentication that quietly breaks rotation. Understand the middleware model and it all falls into place.

This is the Scrapy entry alongside residential proxies with Python, which covers requests and httpx. Scrapy’s downloader-middleware pipeline is a different beast, so it gets its own treatment here.

Everything below uses Shifter’s residential gateway: one endpoint, p.shifter.io:443, with all targeting encoded in the username. Swap the host and credentials for another provider; the shape is the same.

The gateway model in one paragraph

The proxy username carries your authentication and your targeting. You do not switch endpoints to change country or session, you change the username string:

customer-USERNAME-country-us-sid-abc123-ttl-600

country-us targets the United States, sid pins a sticky session, ttl holds that IP for N seconds. Omit sid/ttl and every new connection rotates. The password stays constant. In Scrapy, that username becomes the Proxy-Authorization header, and rotating identity means changing it per request.

How Scrapy handles proxies

Scrapy routes every request’s proxy through the built-in HttpProxyMiddleware, which reads request.meta['proxy']. The naive setup is to put credentials inline in that URL:

# The tempting one-liner. It works, until you rotate.
request.meta['proxy'] = 'http://customer-USER-country-us:PASS@p.shifter.io:443'

This works for a single fixed identity. It breaks the moment you try to rotate, and the reason is the trap worth knowing.

The trap: Proxy-Authorization is cached

When HttpProxyMiddleware sees credentials in the proxy URL, it base64-encodes them into a Proxy-Authorization header and, crucially, caches that header on the request. If a request is later retried or redirected and its meta['proxy'] changes to a different identity, the middleware does not always recompute the header, so the request goes out with a stale Proxy-Authorization for the previous username. On a gateway where the username carries your geo and session, that means your rotation silently does not rotate: you change the username in meta['proxy'], but the request still authenticates as the old one.

The fix is to stop putting credentials in the proxy URL at all. Set the proxy host without userinfo, and set the Proxy-Authorization header yourself, explicitly, on every request. That is exactly what a custom middleware is for.

A custom rotation middleware

Put the host in meta['proxy'] with no credentials, and compute the auth header per request from the identity you want. Because targeting lives in the username, choosing a country and session is just building the right username.

import os
from w3lib.http import basic_auth_header

class ShifterProxyMiddleware:
    def __init__(self):
        self.user = os.environ['SHIFTER_USER']
        self.password = os.environ['SHIFTER_PASS']
        self.endpoint = 'http://p.shifter.io:443'   # host only, no credentials

    def process_request(self, request, spider):
        country = request.meta.get('country', 'us')
        sid = request.meta.get('sid')               # set for a sticky session, omit to rotate
        username = f"{self.user}-country-{country}" + (f"-sid-{sid}-ttl-600" if sid else "")
        request.meta['proxy'] = self.endpoint
        request.headers['Proxy-Authorization'] = basic_auth_header(username, self.password)

Enable it, and let it run before the built-in proxy middleware so the header you set is the one that ships:

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.ShifterProxyMiddleware': 350,   # before HttpProxyMiddleware (750)
}

Now every request carries its own freshly computed Proxy-Authorization, so changing country or sid in a request’s meta actually changes the identity. Set sid on the requests that belong to one logical unit of work so they share an IP, and leave it off to rotate per connection (sticky vs rotating covers the distinction). Mapping work to identities this way is the load-balancing pattern in Scrapy form.

Rotate on retry, not just on schedule

Scrapy’s RetryMiddleware already retries timeouts and 5xxs, but by default it retries with the same identity, which is pointless if the reason for the failure was that identity getting blocked. The high-value move is to rotate identity specifically when a request fails or comes back challenged. In your middleware, detect a soft block or a 403/429 and re-schedule the request with a new identity:

    def process_response(self, request, response, spider):
        if response.status in (403, 429) or looks_blocked(response):
            new = request.copy()
            new.meta.pop('sid', None)          # drop the burned session -> fresh IP
            new.dont_filter = True
            return new                          # retry through a new identity
        return response

Detecting the soft block is its own discipline, a 200 can still be a block page, so pair this with the checks in detecting blocked or fake content. Treating a challenged response as a signal to rotate, rather than accepting it, is what keeps a long crawl alive.

Use Scrapy’s politeness knobs

Scrapy gives you the rate-limiting controls that a hand-rolled scraper has to build, and with a proxy fleet they matter more, not less. Cap concurrency per domain so one target is not hammered, add a delay, and turn on AutoThrottle to adapt to the site’s responses:

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8      # per-target cap, the one that matters
DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = True
RETRY_ENABLED = True
RETRY_TIMES = 3

Per-domain concurrency is the knob that keeps you from turning a proxy pool into a distributed hammer. More parallelism past a target’s tolerance buys blocks, not throughput (how to avoid getting blocked and scraping responsibly both apply), and AutoThrottle backing off on slow responses is exactly the restraint a healthy long-running crawl needs.

Verify you are actually on the proxy

Point a spider at an IP-echo endpoint and check the exit IP before trusting a run:

def start_requests(self):
    yield scrapy.Request('http://ip-api.com/json',
                         meta={'country': 'us'},
                         callback=self.parse)   # expect a US residential IP

Your own IP means the middleware is not applying, or is ordered after HttpProxyMiddleware. A wall of timeouts means the auth header is wrong or missing. Both are covered in the timeout diagnosis guide.

FAQ

Why does my proxy rotation not actually rotate in Scrapy? Almost certainly the Proxy-Authorization caching trap: you put credentials in the proxy URL, and HttpProxyMiddleware cached the auth header, so when you change meta['proxy'] on a retry the request still sends the old credentials. Set the host without credentials and compute the Proxy-Authorization header yourself in a middleware, per request.

Where do the targeting flags go? Into the username, which becomes the Proxy-Authorization. A custom middleware builds customer-USER-country-<cc>-sid-<id>-ttl-<sec> from per-request meta, so choosing geo and session is just setting country and sid on the request.

How do I give a specific request a sticky session? Set a stable sid in that request’s meta and reuse it across the requests that belong together; omit sid to rotate on every connection. The middleware turns that into the right username.

Should I rotate on every request or on retry? Both have a place. Rotate per logical unit of work for normal traffic, and additionally force a fresh identity when a request comes back blocked or rate-limited, so a burned IP is not retried as itself.

Do I still need DOWNLOAD_DELAY and AutoThrottle with rotating proxies? Yes. Rotation spreads load across IPs, but per-domain concurrency, delay, and AutoThrottle keep you from overwhelming a single target regardless of how many IPs you have. Politeness and rotation solve different problems.

The bottom line

Scrapy plus residential proxies is powerful once you route around its one trap: do not put credentials in the proxy URL, because the cached Proxy-Authorization silently defeats rotation. Instead, write a small downloader middleware that sets the host in meta['proxy'] and computes the Proxy-Authorization header per request from the identity you want, rotate that identity per logical unit of work and again on any blocked or rate-limited response, and lean on Scrapy’s per-domain concurrency and AutoThrottle to stay polite.

Do that and Scrapy’s scheduler, retries, and pipelines work with your proxy layer instead of against it. Point the crawl at the residential gateway, and remember that pool quality decides how often you retry at all (IP reputation). The pricing page has the per-GB plans to test it against your own targets.