Start Free TrialLog In
Integration

Use Shifter with Scrapy

Wire Shifter's residential and ISP proxies into any Scrapy spider via a tiny downloader middleware. Per-request rotation, sticky sessions, and per-spider geo-targeting — all in 20 lines of Python.

Quick Start

Install

pip install scrapy

Basic Usage

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ShifterProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 360,
}

# middlewares.py
class ShifterProxyMiddleware:
    PROXY = (
        "customer-USERNAME-country-us-sid-123ABC:"
        "PASSWORD@p.shifter.io:443"
    )

    def process_request(self, request, spider):
        request.meta["proxy"] = self.PROXY

# Run as usual:
# scrapy crawl my_spider

Features

Plug into Scrapy's standard downloader-middleware pipeline — no fork or patched runtime
Per-request rotation by default, with `sid` for sticky sessions and `ttl-N` for timed pins of N seconds
Drop-in compatible with scrapy-playwright, scrapy-splash, and scrapy-rotating-proxies
Geo-targeting in 195+ countries via username parameters — country, region, city, ASN
Works with Scrapy Cloud, Scrapyd, GitHub Actions, Airflow, and any orchestration layer
Compatible with Scrapy 2.x and Python 3.7+ — supports both sync and async callbacks

Examples

Downloader Middleware (Sticky Session)

The standard way to plug a proxy into Scrapy. Add a `sid` to the username and every request from the spider will share one residential IP. Add `country-uk-city-london` to geo-target.

# myproject/middlewares.py
import secrets

class ShifterProxyMiddleware:
    """Routes every Scrapy request through Shifter's residential pool."""

    def __init__(self, country="us", city=None, ttl=300):
        self.sid = secrets.token_hex(4)
        parts = [
            "customer-USERNAME",
            f"country-{country}",
        ]
        if city:
            parts.append(f"city-{city}")
        parts.append(f"sid-{self.sid}")
        parts.append(f"ttl-{ttl}")
        username = "-".join(parts)

        self.proxy_url = f"http://{username}:PASSWORD@p.shifter.io:443"

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        return cls(
            country=s.get("SHIFTER_COUNTRY", "us"),
            city=s.get("SHIFTER_CITY"),
            ttl=s.getint("SHIFTER_TTL", 300),
        )

    def process_request(self, request, spider):
        request.meta["proxy"] = self.proxy_url

# myproject/settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ShifterProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 360,
}

SHIFTER_COUNTRY = "uk"
SHIFTER_CITY    = "london"

Per-Request Rotation

Don't set a sid — let the gateway rotate IPs on every request. Useful for high-volume scraping of paginated targets where each page should look like a different visitor.

# myproject/middlewares.py
import secrets

class ShifterRotatingMiddleware:
    """Rotates the residential IP on every Scrapy request."""

    PROXY_HOST = "p.shifter.io:443"

    def process_request(self, request, spider):
        # Unique sid per request -> guaranteed new IP for every fetch
        unique_sid = secrets.token_hex(6)
        username   = (
            f"customer-USERNAME-country-{spider.country}"
            f"-sid-{unique_sid}"
        )
        request.meta["proxy"] = (
            f"http://{username}:PASSWORD@{self.PROXY_HOST}"
        )

# myproject/spiders/products.py
import scrapy

class ProductsSpider(scrapy.Spider):
    name    = "products"
    country = "us"  # consumed by the middleware

    custom_settings = {
        "DOWNLOADER_MIDDLEWARES": {
            "myproject.middlewares.ShifterRotatingMiddleware": 350,
        },
        "CONCURRENT_REQUESTS": 32,
    }

    start_urls = [
        f"https://example.com/products?page={i}" for i in range(1, 100)
    ]

    def parse(self, response):
        for card in response.css(".product-card"):
            yield {
                "title": card.css("h2::text").get(),
                "price": card.css(".price::text").get(),
                "url":   response.urljoin(card.css("a::attr(href)").get()),
            }

Per-Country Spiders (concurrent geo-scraping)

Build one spider class and parameterize the country at run time. Run multiple instances in parallel — each with its own residential IP pool.

# scrapy crawl localized -a country=uk
# scrapy crawl localized -a country=de
# scrapy crawl localized -a country=jp

import scrapy

class LocalizedSpider(scrapy.Spider):
    name = "localized"

    def __init__(self, country="us", *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.country = country
        self.start_urls = [
            f"https://www.example.com/{country}/products",
        ]

    def start_requests(self):
        proxy = (
            f"customer-USERNAME-country-{self.country}-sid-{self.country}-batch:"
            f"PASSWORD@p.shifter.io:443"
        )
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": proxy}, callback=self.parse)

    def parse(self, response):
        for product in response.css(".product"):
            yield {
                "country": self.country,
                "title":   product.css("h2::text").get(),
                "price":   product.css(".price::text").get(),
            }

Scrapy + scrapy-playwright (JS-rendered pages)

When the target needs JavaScript, swap the downloader for scrapy-playwright. Pass the proxy on the launch options — Scrapy still handles scheduling and pipelines.

# pip install scrapy-playwright
# playwright install chromium

# settings.py
DOWNLOAD_HANDLERS = {
    "http":  "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
    "proxy": {
        "server":   "http://p.shifter.io:443",
        "username": "customer-USERNAME-country-fr-sid-789GHI",
        "password": "PASSWORD",
    },
}

# spider.py
import scrapy

class JsHeavySpider(scrapy.Spider):
    name = "js_heavy"
    start_urls = ["https://app.example.com/dashboard"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"playwright": True, "playwright_include_page": True},
                callback=self.parse,
            )

    async def parse(self, response):
        page = response.meta["playwright_page"]
        await page.wait_for_selector(".widget")
        widgets = await page.query_selector_all(".widget")
        for w in widgets:
            yield {"label": await w.text_content()}
        await page.close()
FAQ

Frequently asked questions

Common questions about using Shifter with Scrapy.

Write a tiny downloader middleware that sets `request.meta['proxy']` to your Shifter URL, then register it in DOWNLOADER_MIDDLEWARES with a priority lower than 750 (so it runs before HttpProxyMiddleware). Twenty lines of Python — no SDK required.

Generate a fresh sid in process_request — for example `secrets.token_hex(6)` — and embed it in the proxy username. Each request will get a different sid and therefore a different residential IP from Shifter's gateway. No external scrapy-rotating-proxies dependency required.

Use a fixed sid for the entire crawl run. Generate it once at the spider's start (or in the middleware constructor) and reuse it across every request. Add `ttl-N` to extend the IP lifetime to N seconds.

Yes. Pass a `-a country=uk` argument when running a spider, expose it as a setting in custom_settings, or read it inside your middleware. Build the proxy URL with `country-uk` in the username — every request from that spider goes through UK residentials.

Yes. Configure the proxy in PLAYWRIGHT_LAUNCH_OPTIONS — `server`, `username`, `password`. scrapy-playwright forwards them to Playwright's launch call, which handles the basic-auth challenge in headless mode without an extension.

Yes. Scrapy Cloud projects are just Scrapy spiders — your downloader middleware and Shifter proxy URL ship inside your project tarball. Add the credentials as Scrapy Cloud project settings (or env vars) so they're not committed to source.
Get started

Start Using Shifter with Scrapy

Plug Shifter's 205M+ residential and ISP proxies into your Scrapy spiders via a 20-line middleware. Per-request rotation, sticky sessions, and full scrapy-playwright support.

Try Shifter for FreeSet up in minutes. Cancel anytime.