How to Use Residential Proxies in Selenium (Including Authenticated Proxies)

Selenium is the most widely deployed browser-automation tool there is, and for scraping a JavaScript-heavy target it does the job. But it has one long-standing gap that trips up almost everyone the first time they add a residential proxy: setting the proxy host is trivial, and providing a username and password is not, because Selenium has no built-in way to do it. Point Chrome at an authenticated proxy and it pops up a native 407 login dialog that Selenium cannot fill, and your script hangs. Here is how to get past that, three ways.

This sits alongside the other browser guides, residential proxies with Playwright and in Puppeteer, both of which handle proxy auth natively. If you do not need a full browser, proxies in Python with a plain HTTP client is simpler still.

Everything below uses Shifter’s residential gateway: one endpoint, p.shifter.io:443, with all targeting encoded in the username. Swap the host and credentials for another provider; the shape is the same.

The gateway model in one paragraph

The proxy username carries your authentication and your targeting. You do not switch endpoints to change country or session, you change the username string:

customer-USERNAME-country-us-sid-abc123-ttl-600

country-us targets the United States, sid pins a sticky session, ttl holds that IP for N seconds. Omit sid/ttl and every new connection rotates. The password stays constant. The whole problem in Selenium is getting that username and password to the proxy.

The core problem

Setting the host is the easy half. You pass --proxy-server in Chrome options exactly as you would elsewhere:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=http://p.shifter.io:443')   # host only
driver = webdriver.Chrome(options=options)

That works for an IP-whitelisted proxy with no credentials. But the gateway is username-and-password authenticated, and Chromium will not read credentials from that flag, so the first navigation stalls on a 407 auth prompt Selenium cannot dismiss. You need one of three ways to answer that challenge.

Approach 1: Selenium Wire (the easy one)

Selenium Wire extends Selenium and accepts proxy credentials directly, handling the auth for you. It is the least-friction option and what most Python scrapers reach for.

from seleniumwire import webdriver   # pip install selenium-wire
import os

user = os.environ['SHIFTER_USER'] + '-country-us'   # targeting in the username
pw = os.environ['SHIFTER_PASS']

seleniumwire_options = {
    'proxy': {
        'http':  f'http://{user}:{pw}@p.shifter.io:443',
        'https': f'http://{user}:{pw}@p.shifter.io:443',
        'no_proxy': 'localhost,127.0.0.1',
    }
}

driver = webdriver.Chrome(seleniumwire_options=seleniumwire_options)
driver.get('https://api.ipify.org')
print(driver.page_source)   # a US residential IP
driver.quit()

The credentials, including the targeting flags in the username, go in the proxy config and Selenium Wire deals with the 407 transparently. It also lets you change the proxy at runtime by reassigning driver.proxy, which is handy for rotation without relaunching Chrome.

Approach 2: a credentials extension (vanilla Selenium, no extra library)

If you want to stay on plain Selenium, the classic technique is to load a tiny Chrome extension that answers the auth challenge with your credentials. You build it on the fly and pass it with add_extension.

# manifest.json declares proxy + auth permissions; background.js supplies creds.
background_js = """
chrome.webRequest.onAuthRequired.addListener(
  () => ({ authCredentials: { username: USER, password: PASS } }),
  { urls: ['<all_urls>'] },
  ['blocking']
);
""".replace('USER', repr(user)).replace('PASS', repr(pw))
# zip manifest.json + background.js, then:
options.add_extension('proxy_auth.zip')

This keeps you dependency-free and works in any Selenium language binding, since the extension does the work. The caveat: the blocking onAuthRequired pattern above is a Manifest V2 technique, and Chrome is phasing MV2 out in favor of MV3, so on current Chrome this approach is more fragile than it used to be. If you are starting fresh, prefer Selenium Wire or the CDP route below.

Approach 3: CDP in Selenium 4

Selenium 4 exposes the Chrome DevTools Protocol, and you can answer proxy auth through the Fetch domain by handling Fetch.authRequired and continuing the request with credentials. It is native to modern Selenium and needs no extra package, but it is fiddly to wire up by hand, essentially reimplementing what Selenium Wire already wraps. Reach for it when you want zero third-party dependencies and are comfortable with CDP; otherwise Selenium Wire saves you the trouble.

Rotating geo and sessions

Because targeting lives in the username, a different identity is a different username, and in Selenium the proxy is set at the browser level. That means rotation happens per driver, not per tab. Two practical patterns: with Selenium Wire, reassign driver.proxy at runtime to swap the username between units of work; with the extension or CDP approach, run a driver per identity and pool them. Either way, give each logical unit of work its own sid and rotate between units, not mid-session (sticky vs rotating covers the distinction), and map work to identities as the load-balancing post describes.

Reuse the driver, and bound concurrency

Launching Chrome is expensive, a fresh driver per request pays real startup time and memory every time, the overhead the latency guide exists to remove. Start a driver (or a small pool of them) and reuse it across requests. And because each driver is a full browser holding real memory, you cannot run thousands, keep a bounded pool and cap in-flight work per target host so one fragile site is not hammered while a permissive one is starved. More parallelism past a target’s tolerance buys blocks and out-of-memory crashes, not throughput (how to avoid getting blocked).

The browser is only half of not getting blocked

A residential IP handles the network half of looking human, but Selenium is still driving an automated browser, and sites fingerprint that too, navigator.webdriver, automation flags, and headless quirks. A clean IP with good reputation keeps you out of many challenges, but it does not disguise an obviously automated browser. Keep the user-agent and viewport realistic, drive the page at a human pace, and remember that the mistakes that trigger detection apply to the browser layer as much as the IP layer. The two have to line up.

Verify you are actually on the proxy

Before you benchmark or debug anything else, confirm the exit IP from inside the browser:

driver.get('http://ip-api.com/json')
print(driver.find_element('tag name', 'body').text)   # expect the targeted country

Your own IP means the proxy is not applied. A hang on a login dialog means the auth step (Selenium Wire, extension, or CDP) is missing or misconfigured. A general hang means local egress is blocked. All three are covered in the timeout diagnosis guide.

FAQ

Why does Selenium hang on a proxy login popup? Chromium raises a native 407 authentication dialog for an authenticated proxy, and Selenium cannot interact with native browser dialogs. You have to answer the challenge another way: Selenium Wire, a credentials extension, or CDP’s Fetch.authRequired. Setting --proxy-server alone only provides the host, not the credentials.

Can I put user:pass@host in --proxy-server? No. Chromium does not read credentials from the --proxy-server flag. Provide the host there and supply the username and password through one of the three approaches above. Because the gateway encodes targeting in the username, the full username (with -country-...) is what you pass as the proxy username.

Do I have to use Selenium Wire? No, but it is the simplest path in Python. The dependency-free alternatives are a credentials extension (works in any language binding, though the classic MV2 pattern is being phased out) or CDP in Selenium 4 (native but more work to wire up).

How do I rotate IPs in Selenium? Vary the proxy username, which changes the identity through the same gateway. In Selenium Wire you can reassign driver.proxy at runtime; otherwise run one driver per identity and pool them. Omit the sid in the username to rotate on every new connection.

Selenium, Playwright, or Puppeteer? Playwright and Puppeteer both take proxy credentials natively, so they avoid this whole dance; Selenium needs one of the workarounds here. Selenium is still a fine choice if it is what your stack already uses; if you are starting fresh and want painless proxy auth, the other two are smoother.

The bottom line

Selenium plus residential proxies works well once you solve the one gap it has: it will take the proxy host in --proxy-server, but it needs help to provide credentials for an authenticated proxy. Use Selenium Wire for the least friction, a credentials extension if you want to stay dependency-free, or CDP in Selenium 4 for a native route. Then rotate geo and sessions by varying the username, reuse a long-lived driver, bound your concurrency because each one is a real browser, and keep the browser fingerprint as human as the IP.

Get that right and Selenium handles the interactive, JavaScript-heavy targets that plain HTTP clients cannot. Point it at the residential gateway, and remember that pool quality decides how often you are challenged at all (IP reputation). The pricing page has the per-GB plans to test it against your own targets.