Start Free TrialLog In
Integration

Use Shifter with Python

Integrate Shifter's residential and ISP proxies into your Python scripts in minutes. Works seamlessly with requests, aiohttp, Scrapy, Selenium, and every major Python HTTP library.

Quick Start

Install

pip install requests

Basic Usage

import requests

proxy_url = "customer-USERNAME-country-us-sid-123ABC:PASSWORD@p.shifter.io:443"

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = requests.get("https://ipinfo.io/json", proxies=proxies)
print(response.json())
# {"ip": "154.16.xxx.xxx", "city": "New York", "country": "US", ...}

Features

Works with all Python HTTP libraries including requests, aiohttp, httpx, and urllib3
Automatic IP rotation per request, per session (sid), or via configurable sticky sessions (ttl)
Full support for both HTTP and SOCKS5 proxy protocols
Compatible with Python 3.7+ and all major scraping frameworks like Scrapy and BeautifulSoup
Geo-targeting support, select proxies from 195+ countries via simple username parameters
No additional SDK required, configure Shifter using standard proxy environment variables or inline settings

Examples

Basic Requests

The simplest way to use Shifter proxies with Python's most popular HTTP library. Set country, region, city, or ASN directly in the username and pass a `sid` to keep the same IP across requests.

import requests

PROXY_USER = "customer-USERNAME-country-us-sid-123ABC"
PROXY_PASS = "PASSWORD"
PROXY_HOST = "p.shifter.io"
PROXY_PORT = "443"

proxies = {
    "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
    "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}

response = requests.get(
    "https://example.com",
    proxies=proxies,
    headers=headers,
    timeout=30,
)

print(f"Status: {response.status_code}")
print(f"Content length: {len(response.text)}")

Async with aiohttp + Sticky Sessions

Use aiohttp for high-performance asynchronous scraping. Add `sid-XXX` to the username to keep the same IP across requests, or `ttl-N` for a timed sticky session of N seconds.

import aiohttp
import asyncio
import uuid

# sid pins the same IP for every request that shares this session id;
# ttl-300 keeps that IP for up to 300 seconds, then rotates.
session_id = uuid.uuid4().hex[:8]
PROXY_URL = (
    f"customer-USERNAME-country-us-sid-{session_id}-ttl-300:"
    f"PASSWORD@p.shifter.io:443"
)

async def fetch(session, url):
    async with session.get(
        url, proxy=PROXY_URL, timeout=aiohttp.ClientTimeout(total=30)
    ) as response:
        return await response.text()

async def main():
    urls = [
        "https://example.com/login",
        "https://example.com/dashboard",
        "https://example.com/orders",
    ]

    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, url) for url in urls]
        results = await asyncio.gather(*tasks, return_exceptions=True)

        for url, result in zip(urls, results):
            if isinstance(result, Exception):
                print(f"Error fetching {url}: {result}")
            else:
                print(f"Fetched {url}: {len(result)} bytes")

asyncio.run(main())

Scrapy Middleware

Integrate Shifter proxies into your Scrapy spiders with a custom downloader middleware. Geo-target by adding selectors like `country-uk`, `region-bavaria`, `city-london`, or `asn-7922` to the username, and pin the session with a `sid`.

# middlewares.py
class ShifterProxyMiddleware:
    PROXY_USER = "customer-USERNAME-country-uk-sid-456DEF"
    PROXY_PASS = "PASSWORD"
    PROXY_HOST = "p.shifter.io"
    PROXY_PORT = "443"

    def process_request(self, request, spider):
        request.meta["proxy"] = (
            f"http://{self.PROXY_USER}:{self.PROXY_PASS}"
            f"@{self.PROXY_HOST}:{self.PROXY_PORT}"
        )

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ShifterProxyMiddleware": 350,
}

# spider.py
import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.co.uk/products"]

    def parse(self, response):
        for product in response.css(".product-card"):
            yield {
                "title": product.css("h2::text").get(),
                "price": product.css(".price::text").get(),
                "url": product.css("a::attr(href)").get(),
            }

        next_page = response.css("a.next-page::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Selenium WebDriver

Route Selenium browser automation through Shifter proxies. Ideal for scraping JavaScript-rendered pages. Combine city-level targeting with a `sid` to keep the same IP for the entire browser session.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

PROXY_HOST = "p.shifter.io"
PROXY_PORT = "443"
PROXY_USER = "customer-USERNAME-country-us-city-newyork-sid-789GHI"
PROXY_PASS = "PASSWORD"

chrome_options = Options()
chrome_options.add_argument(
    f"--proxy-server=http://{PROXY_HOST}:{PROXY_PORT}"
)

driver = webdriver.Chrome(options=chrome_options)

# Handle proxy authentication via browser extension or seleniumwire if needed
driver.get("https://example.com")

# Wait for dynamic content to load
wait = WebDriverWait(driver, 10)
element = wait.until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".content"))
)

print(f"Page title: {driver.title}")
print(f"Content: {element.text[:200]}")

driver.quit()
FAQ

Frequently asked questions

Common questions about using Shifter with Python.

Pass a proxies dictionary to any requests method with your Shifter proxy URL. The format is: proxies = {"http": "customer-USERNAME-country-us-sid-123ABC:PASSWORD@p.shifter.io:443", "https": "customer-USERNAME-country-us-sid-123ABC:PASSWORD@p.shifter.io:443"}. Then call requests.get(url, proxies=proxies).

Yes. Shifter proxies work with all async Python HTTP libraries including aiohttp, httpx, and Twisted. Pass the proxy URL using the library's proxy parameter. For aiohttp, use the proxy argument in session.get(). For httpx, configure proxies in the Client constructor.

Shifter's rotating proxy gateway handles IP rotation automatically. Rotate per request by default, or control it via session ID (sid) or time-to-live (ttl). No additional code is needed.

Yes. You can integrate Shifter with Scrapy by creating a simple downloader middleware that sets request.meta['proxy'] to your Shifter proxy URL. Add the middleware to your DOWNLOADER_MIDDLEWARES setting and every Scrapy request will automatically route through Shifter's residential proxies.

Yes. Configure Selenium's Chrome or Firefox WebDriver to use Shifter as an HTTP proxy via browser options. For Chrome, use --proxy-server flag in ChromeOptions. For authenticated proxies, you can use a browser extension or wire protocol to pass credentials.

Shifter proxies work with any Python version that supports HTTP proxies, which includes Python 3.7 and above. The proxy is configured at the HTTP level, so it's compatible with all Python HTTP libraries regardless of version. No special SDK or package is required beyond your preferred HTTP client.
Get started

Start Using Shifter with Python

Integrate Shifter's 205M+ residential proxies into your Python scripts in under 5 minutes. Flexible IP rotation, geo-targeting, and full compatibility with every major Python library.

Try Shifter for FreeSet up in minutes. Cancel anytime.