开始免费试用登录
集成

将Shifter与以下工具配合使用 Scrapy

通过简短的下载器中间件，将 Shifter 的住宅代理和 ISP 代理接入任意 Scrapy 爬虫。每请求轮换、粘性会话和按爬虫地理定向——仅需 20 行 Python。

快速入门

安装

pip install scrapy

基本用法

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ShifterProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 360,
}

# middlewares.py
class ShifterProxyMiddleware:
    PROXY = (
        "customer-USERNAME-country-us-sid-123ABC:"
        "PASSWORD@p.shifter.io:443"
    )

    def process_request(self, request, spider):
        request.meta["proxy"] = self.PROXY

# Run as usual:
# scrapy crawl my_spider

功能特性

接入 Scrapy 标准下载器中间件管道——无需分叉或修补运行时
默认按请求轮换，使用`sid`实现粘性会话，使用`ttl-N`实现N秒定时固定
与 scrapy-playwright、scrapy-splash 和 scrapy-rotating-proxies 即插即用兼容
通过用户名参数在 195+ 个国家/地区进行地理定向 — country、region、city、ASN
支持 Scrapy Cloud、Scrapyd、GitHub Actions、Airflow 及任何编排层
兼容 Scrapy 2.x 和 Python 3.7+——同时支持同步和异步回调

示例

下载器中间件（粘性会话）

将代理接入 Scrapy 的标准方式。在用户名中添加 `sid`，爬虫的每个请求将共享同一个住宅 IP。添加 `country-uk-city-london` 可进行地理定向。

# myproject/middlewares.py
import secrets

class ShifterProxyMiddleware:
    """Routes every Scrapy request through Shifter's residential pool."""

    def __init__(self, country="us", city=None, ttl=300):
        self.sid = secrets.token_hex(4)
        parts = [
            "customer-USERNAME",
            f"country-{country}",
        ]
        if city:
            parts.append(f"city-{city}")
        parts.append(f"sid-{self.sid}")
        parts.append(f"ttl-{ttl}")
        username = "-".join(parts)

        self.proxy_url = f"http://{username}:PASSWORD@p.shifter.io:443"

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        return cls(
            country=s.get("SHIFTER_COUNTRY", "us"),
            city=s.get("SHIFTER_CITY"),
            ttl=s.getint("SHIFTER_TTL", 300),
        )

    def process_request(self, request, spider):
        request.meta["proxy"] = self.proxy_url

# myproject/settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ShifterProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 360,
}

SHIFTER_COUNTRY = "uk"
SHIFTER_CITY    = "london"

按请求轮换

不设置 sid——让网关在每次请求时轮换 IP。适用于对分页目标进行大批量抓取，使每个页面看起来来自不同访客。

# myproject/middlewares.py
import secrets

class ShifterRotatingMiddleware:
    """Rotates the residential IP on every Scrapy request."""

    PROXY_HOST = "p.shifter.io:443"

    def process_request(self, request, spider):
        # Unique sid per request -> guaranteed new IP for every fetch
        unique_sid = secrets.token_hex(6)
        username   = (
            f"customer-USERNAME-country-{spider.country}"
            f"-sid-{unique_sid}"
        )
        request.meta["proxy"] = (
            f"http://{username}:PASSWORD@{self.PROXY_HOST}"
        )

# myproject/spiders/products.py
import scrapy

class ProductsSpider(scrapy.Spider):
    name    = "products"
    country = "us"  # consumed by the middleware

    custom_settings = {
        "DOWNLOADER_MIDDLEWARES": {
            "myproject.middlewares.ShifterRotatingMiddleware": 350,
        },
        "CONCURRENT_REQUESTS": 32,
    }

    start_urls = [
        f"https://example.com/products?page={i}" for i in range(1, 100)
    ]

    def parse(self, response):
        for card in response.css(".product-card"):
            yield {
                "title": card.css("h2::text").get(),
                "price": card.css(".price::text").get(),
                "url":   response.urljoin(card.css("a::attr(href)").get()),
            }

按国家划分的爬虫（并发地理抓取）

构建一个爬虫类，并在运行时参数化国家。并行运行多个实例——每个实例拥有独立的住宅 IP 池。

# scrapy crawl localized -a country=uk
# scrapy crawl localized -a country=de
# scrapy crawl localized -a country=jp

import scrapy

class LocalizedSpider(scrapy.Spider):
    name = "localized"

    def __init__(self, country="us", *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.country = country
        self.start_urls = [
            f"https://www.example.com/{country}/products",
        ]

    def start_requests(self):
        proxy = (
            f"customer-USERNAME-country-{self.country}-sid-{self.country}-batch:"
            f"PASSWORD@p.shifter.io:443"
        )
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": proxy}, callback=self.parse)

    def parse(self, response):
        for product in response.css(".product"):
            yield {
                "country": self.country,
                "title":   product.css("h2::text").get(),
                "price":   product.css(".price::text").get(),
            }

Scrapy + scrapy-playwright（JS 渲染页面）

当目标页面需要 JavaScript 时，将下载器替换为 scrapy-playwright。在启动选项中传入代理——Scrapy 仍负责调度和管道处理。

# pip install scrapy-playwright
# playwright install chromium

# settings.py
DOWNLOAD_HANDLERS = {
    "http":  "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
    "proxy": {
        "server":   "http://p.shifter.io:443",
        "username": "customer-USERNAME-country-fr-sid-789GHI",
        "password": "PASSWORD",
    },
}

# spider.py
import scrapy

class JsHeavySpider(scrapy.Spider):
    name = "js_heavy"
    start_urls = ["https://app.example.com/dashboard"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"playwright": True, "playwright_include_page": True},
                callback=self.parse,
            )

    async def parse(self, response):
        page = response.meta["playwright_page"]
        await page.wait_for_selector(".widget")
        widgets = await page.query_selector_all(".widget")
        for w in widgets:
            yield {"label": await w.text_content()}
        await page.close()
常见问题

常见问题

关于将 Shifter 与 Scrapy 搭配使用的常见问题。

编写一个简短的下载器中间件，将 `request.meta['proxy']` 设置为你的 Shifter URL，然后在 DOWNLOADER_MIDDLEWARES 中以低于 750 的优先级注册（使其在 HttpProxyMiddleware 之前运行）。仅需 20 行 Python，无需 SDK。
立即开始

开始将Shifter与以下工具配合使用 Scrapy

通过 20 行中间件将 Shifter 的 205M+ 住宅和 ISP 代理接入您的 Scrapy 爬虫。支持按请求轮换、粘性会话以及完整的 scrapy-playwright 集成。

免费试用 Shifter几分钟内完成设置，随时可取消。