Residential Proxies for LangChain, CrewAI & AutoGen Agents

An agent framework makes it easy to give a model a web tool. It does not make the web easy to reach. The first time a LangChain, CrewAI or AutoGen agent fetches pages at any volume, it starts collecting 403s, CAPTCHA pages and content for the wrong country, and the model happily summarises all of it as if it were real. Routing the agent’s web tools through proxies built for AI agents fixes the access side, but only if you wire them in the right place.

This guide is the wiring. It shows how to send each framework’s web tools through a residential proxy, how to keep the model API off the proxy, and how to pick sticky or rotating sessions per task. If you are still choosing a proxy type, start with our comparison of the best proxies for AI agents that browse the web; this post assumes you have picked one.

Rule one: proxy the tools, not the process

The quickest way to add a proxy is to export HTTPS_PROXY and restart. Do not do that in an agent process. The OpenAI, Anthropic and most other model SDKs read the same environment variables, so every model call would also go through the residential network. That adds latency to every reasoning step, spends proxy traffic on tokens, and gives you nothing in return, because model APIs do not block you by IP.

Set the proxy explicitly on the HTTP client or browser each tool uses. Everything else stays on your normal connection. If a library only reads environment variables, add the model hosts to NO_PROXY.

The proxy URL

With the Shifter gateway, targeting and sessions live in the username. One endpoint, p.shifter.io:443, covers every country:

import os, uuid

USER = os.environ["SHIFTER_USER"]
PASSWORD = os.environ["SHIFTER_PASS"]

def proxy_url(country: str = "us", session: str | None = None, ttl: int = 600) -> str:
    """Rotating by default; pass a session id to keep one exit IP for a task."""
    username = f"{USER}-country-{country}"
    if session:
        username += f"-sid-{session}-ttl-{ttl}"
    return f"http://{username}:{PASSWORD}@p.shifter.io:443"

def new_session() -> str:
    return uuid.uuid4().hex[:12]

Every example below reuses this helper.

LangChain

Most LangChain agents read the web in one of two ways: a document loader for retrieval, or a browser toolkit for agents that click and type.

Document loaders. WebBaseLoader accepts a proxies dictionary and uses it for every page it fetches:

from langchain_community.document_loaders import WebBaseLoader

proxy = proxy_url(country="de")
loader = WebBaseLoader(
    ["https://shop.example/de/product/123"],
    proxies={"http": proxy, "https": proxy},
)
docs = loader.load()

Rotation suits loaders: each page is independent, so a new IP per request spreads the load.

Browser toolkits. The Playwright toolkit drives a real browser. Launch the browser with the proxy yourself and hand it to the toolkit:

from langchain_community.agent_toolkits import PlayWrightBrowserToolkit
from playwright.sync_api import sync_playwright

session = new_session()
playwright = sync_playwright().start()
browser = playwright.chromium.launch(proxy={
    "server": "http://p.shifter.io:443",
    "username": f"{USER}-country-us-sid-{session}-ttl-1800",
    "password": PASSWORD,
})
tools = PlayWrightBrowserToolkit.from_browser(sync_browser=browser).get_tools()

Here the session id matters. A browsing agent navigates, reads, clicks and comes back. If its IP changed between steps, the site would see a visitor jump between cities mid-session, which is exactly what bot defences look for. One session id per agent run keeps a single exit for the whole task.

CrewAI

CrewAI agents use tools, and the cleanest approach is a small custom tool that owns its HTTP client:

import requests
from crewai.tools import BaseTool

class FetchPage(BaseTool):
    name: str = "fetch_page"
    description: str = "Fetch a public web page as seen from a given country."

    def _run(self, url: str, country: str = "us") -> str:
        proxy = proxy_url(country=country)
        r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=30)
        r.raise_for_status()
        return r.text[:20000]

Give the tool to the agents that research, and leave it off the agents that only write or review. A crew with five agents rarely needs five agents on the proxy.

Letting the model choose country is useful for market research crews: the researcher agent can compare what a page shows in the US, Germany and Japan without any extra code. Keep the list of allowed countries in the tool description so the model does not invent codes.

AutoGen

In AutoGen’s AgentChat, a tool is a plain Python function, so the proxy goes into the client inside it. An async client fits AutoGen’s event loop:

import httpx
from autogen_agentchat.agents import AssistantAgent

async def fetch_page(url: str, country: str = "us") -> str:
    """Fetch a public web page through a residential IP in the given country."""
    async with httpx.AsyncClient(proxy=proxy_url(country=country), timeout=30) as client:
        r = await client.get(url, follow_redirects=True)
        r.raise_for_status()
        return r.text[:20000]

researcher = AssistantAgent(
    name="researcher",
    model_client=model_client,  # your model client, not proxied
    tools=[fetch_page],
)

The model client is created separately and never sees the proxy, which is the point of rule one.

Sticky or rotating, per task

The framework does not decide this; the task does. A simple rule of thumb:

Agent task	Session	Why
Fetching many independent pages	Rotating	Spreads requests, no state to keep
Multi-step browsing, forms, carts	Sticky, one id per run	The site expects one visitor
Logged-in workflows over hours	ISP Proxies	One city and ISP for the whole billing period
Comparing countries	Rotating, one country per call	Each call is its own market

For a deeper look at the trade-offs, see our decision framework for rotating proxy IPs. For long logged-in sessions, ISP Proxies keep the same city and ISP for the whole billing period, which suits agents that hold an account open.

Make failures visible to the model

Agents are bad at noticing they were blocked. A CAPTCHA page is just more text to summarise. Two small habits prevent wrong answers:

• Raise on 403 and 429 instead of returning the body. The framework passes the error back to the model, which can retry or report it, rather than reasoning over a block page.
• Retry with a new session, not the same one. If a sticky session hits a block, generate a new session id for the retry. Retrying on the same exit usually gets the same answer.

Cap the retries. Three attempts per URL is plenty; beyond that you are paying for traffic the target has already decided to refuse.

Cost control

Agents can loop. A planner that keeps re-reading the same page will spend proxy traffic as fast as it spends tokens. Truncate page bodies before they reach the model, as the examples above do, cache fetched pages for the length of a run, and log bytes per tool call so a runaway agent shows up on a dashboard before it shows up on an invoice.

FAQ

The bottom line

Give the proxy to the tools, not the process. Use rotation where pages are independent, one sticky session per run where the agent behaves like a visitor, and make blocks loud enough that the model cannot mistake them for content.