Scraping Frameworks With Built-In Proxy Management: What Each One Handles

Almost every scraping framework lets you set a proxy. Far fewer manage proxies for you: rotating them, retiring the blocked ones, and keeping a session on one IP. If you use residential proxies, knowing which part the framework handles and which part the gateway handles saves a lot of glue code.

This guide compares six common tools on three questions: how you set the proxy, whether the tool rotates proxies itself, and whether it ties a session to one proxy. Then it shows the small amount of code each one needs behind a rotating gateway.

The short answer

Tool	Language	How you set the proxy	Rotates on its own	Ties a session to a proxy
Crawlee	Node.js	ProxyConfiguration	Yes: round robin, custom function, tiers	Yes, through its session pool
Scrapy	Python	request.meta["proxy"] or environment variables	No, needs your own middleware	No
Colly	Go	SetProxy or SetProxyFunc	Yes: round-robin switcher	No
Playwright	Node.js, Python, Java, .NET	At launch or per browser context	No	Per context
Puppeteer	Node.js	Launch flag or per browser context	No	Per context
Selenium	Many	Browser-wide option	No	No

Crawlee is the only one that treats proxies as something to manage rather than a setting. The browser tools give you isolation per context, which is the building block for sessions, but the rotation logic is yours.

Crawlee: proxies as a managed resource

Crawlee’s ProxyConfiguration takes a list of proxy URLs and rotates through them “in a round-robin fashion”. Instead of a list you can pass a function that picks the proxy for each request. A third mode, tiered proxies, starts with the cheapest tier (often no proxy at all), moves up a tier when a site blocks the crawler, and drops back down when the lower tier works again.

The part that matters most is sessions. When the crawler runs with its session pool, every session keeps the same proxy: in the docs’ words, later calls “with the same sessionId will always return the same proxyUrl”. When Crawlee decides a session is blocked, it retires it and the next session gets a different proxy. Cookies follow the session, so the IP and the cookies stay together.

Scrapy: one proxy per request, rotation is yours

Scrapy’s HttpProxyMiddleware reads the proxy from request.meta["proxy"], and falls back to the http_proxy and https_proxy environment variables. Credentials can sit in the URL, as in http://username:password@host:port. That is all it does. Picking a different proxy per request, noticing bans and resting dead proxies are left to you or to an extension such as scrapy-rotating-proxies, which reads a list of proxies and treats non-200 or empty responses as bans by default.

Our Scrapy integration guide covers the basic setup, and the post on rotating residential proxies in Scrapy goes deeper into middleware.

Colly: a simple round-robin switcher

Colly, the Go framework, has SetProxy for one proxy and SetProxyFunc for a function that chooses one per request. Its proxy package includes RoundRobinProxySwitcher, which “rotates ProxyURLs on every request” and accepts http, https and socks5 URLs. There is no session concept: if two requests must share an IP, you decide that in your own function.

Playwright and Puppeteer: isolation per context

Browser automation tools do not rotate anything, but they let each browser context use its own proxy, which is how you run many separate sessions in one browser.

• Playwright takes a proxy option with server, username and password, either at launch for the whole browser or on newContext() for one context. See our Playwright integration guide for the full setup.
• Puppeteer sets a proxy with the --proxy-server launch flag, or per context with createBrowserContext({ proxyServer }). Neither takes credentials: you answer the proxy’s login challenge with page.authenticate().

Give each context its own session and you get one IP and one cookie jar per context, which is what a logged-in flow needs.

Selenium: browser-wide, and awkward with passwords

Selenium sets the proxy through browser options, so it applies to the whole browser. With Chrome the usual route is the --proxy-server switch, which has no place for a username and password. The common workarounds are a small browser extension that answers the login challenge, or a local forwarding proxy without a password that passes traffic on to your provider with credentials. Selenium Wire, once the popular fix, was archived in January 2024 and is no longer maintained.

What a rotating gateway changes

Framework rotation was designed for lists of proxies: a hundred IP:port pairs that you cycle, test and retire. A rotating residential gateway works differently. You connect to one host and port, and the gateway chooses the exit IP. That moves most of the work out of your code:

• Rotation happens at the gateway, so a single proxy URL is enough. A round-robin list of one entry is just a setting.
• Sessions are requested in the username. With Shifter you add sid-<id> to keep one exit IP and ttl-<seconds> to say how long to hold it, as explained in sticky vs rotating proxies.
• Location is also part of the username, so a crawler can target a country or city per request without a second proxy list.

So the job of the framework shrinks to one thing: map its own idea of a session to the session ID in the proxy username.

Wiring it up

Crawlee. Pass a function that builds the username from Crawlee’s session ID. Session IDs can contain characters such as underscores, so keep only letters and digits:

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
  newUrlFunction: (sessionId) => {
    const sid = sessionId
      ? `-sid-${String(sessionId).replace(/[^a-zA-Z0-9]/g, '')}-ttl-600`
      : '';
    return `http://customer-USERNAME${sid}:PASSWORD@p.shifter.io:443`;
  },
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  useSessionPool: true,
  persistCookiesPerSession: true,
  async requestHandler({ request, $ }) {
    console.log(request.url, $('title').text());
  },
});

await crawler.run(['https://example.com/']);

When Crawlee retires a blocked session, the next session ID gives a new sticky IP on its own.

Scrapy. A small downloader middleware sets the proxy before Scrapy’s own proxy middleware runs. Requests that carry a session value in meta share one IP; the rest rotate:

# middlewares.py
class GatewayProxyMiddleware:
    def process_request(self, request, spider):
        sid = request.meta.get("session")
        user = "customer-USERNAME"
        if sid:
            user += f"-sid-{sid}-ttl-600"
        request.meta["proxy"] = f"http://{user}:PASSWORD@p.shifter.io:443"

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.GatewayProxyMiddleware": 350,
}

Playwright. One context per session:

const context = await browser.newContext({
  proxy: {
    server: 'http://p.shifter.io:443',
    username: 'customer-USERNAME-sid-cart1-ttl-600',
    password: 'PASSWORD',
  },
});

Which one to choose

Choose the framework for the site and the team, not for its proxy features. Crawlee is the strongest pick when you want blocking, sessions and proxy tiers handled for you in Node.js. Scrapy remains the fastest way to crawl large numbers of plain HTML pages in Python, and its proxy gap is a ten-line middleware. Playwright or Puppeteer are the right tools when pages need a real browser, and contexts give you clean sessions. Selenium works too, with more effort around credentials.

Behind a rotating gateway, all six reduce to the same pattern: one proxy URL, and a session ID in the username wherever requests must share an IP. For more on putting this into a production crawler, see our web scraping use case.

Sources and references

• Crawlee, proxy management guide: crawlee.dev
• Scrapy, downloader middleware reference (HttpProxyMiddleware): docs.scrapy.org
• scrapy-rotating-proxies: github.com
• Colly, proxy package: pkg.go.dev
• Playwright, network and HTTP proxy: playwright.dev
• Puppeteer, BrowserContextOptions: pptr.dev
• Selenium Wire repository (archived 3 January 2024): github.com