Residential proxies are what we sell, so it may seem odd for us to write about when you do not need one. But teams that buy proxies for every job overspend, and teams that blame the proxy for every failure waste weeks. Knowing which sites need a residential IP, which need nothing, and which need something else entirely is the cheapest optimisation in web data collection.
To make that concrete, we ran a small test. On 9 October 2026 we requested 30 public pages, three times each, from a cloud server with a datacenter IP address, then again through Shifter residential proxies with US exits. Both runs used the same simple Python HTTP client. Sixteen of the 30 pages worked from the datacenter IP every time. Three worked only through residential exits. The remaining eleven failed both ways, for reasons a residential IP does not fix.
The test
The targets covered six kinds of site: public APIs and open data, reference and documentation sites, news homepages, e-commerce search pages, travel search pages, and a set of social, jobs and search pages. Each request used the same desktop browser User-Agent and was checked for real content, not just a status code: a response only counted as a success if it contained a marker that appears only in the actual content, such as a search result card. Block pages that return status 200, which are common, therefore counted as failures.
We chose the markers by inspecting saved responses from both networks before the final run, and dropped five candidate pages where we could not tell real content from a login wall or an empty JavaScript shell.
| Category | Site | Datacenter IP | Residential exits | Verdict |
|---|---|---|---|---|
| Public APIs and open data | en.wikipedia.org | 3 of 3 | 3 of 3 | No proxy needed |
| Public APIs and open data | api.github.com | 3 of 3 | 3 of 3 | No proxy needed |
| Public APIs and open data | data.sec.gov | 3 of 3 | 3 of 3 | No proxy needed |
| Public APIs and open data | api.worldbank.org | 3 of 3 | 3 of 3 | No proxy needed |
| Public APIs and open data | ecb.europa.eu | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | docs.python.org | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | developer.mozilla.org | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | rfc-editor.org | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | en.wikipedia.org | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | arxiv.org | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | stackoverflow.com | 0 of 3 | 0 of 3 | Blocked either way |
| News | bbc.com | 3 of 3 | 3 of 3 | No proxy needed |
| News | reuters.com | 0 of 3 | 0 of 3 | Blocked either way |
| News | nytimes.com | 0 of 3 | 0 of 3 | Blocked either way |
| News | theguardian.com | 3 of 3 | 3 of 3 | No proxy needed |
| News | edition.cnn.com | 3 of 3 | 3 of 3 | No proxy needed |
| News | bloomberg.com | 0 of 3 | 0 of 3 | Blocked either way |
| E-commerce search | amazon.com | 3 of 3 | 2 of 3 | No proxy needed |
| E-commerce search | walmart.com | 0 of 3 | 3 of 3 | Residential IP needed |
| E-commerce search | ebay.com | 0 of 3 | 0 of 3 | Blocked either way |
| E-commerce search | bestbuy.com | 0 of 3 | 0 of 3 | Timed out either way |
| E-commerce search | etsy.com | 0 of 3 | 0 of 3 | Blocked either way |
| Travel | booking.com | 0 of 3 | 2 of 3 | Residential IP needed |
| Travel | expedia.com | 0 of 3 | 0 of 3 | Blocked either way |
| Travel | airbnb.com | 3 of 3 | 2 of 3 | No proxy needed |
| Travel | tripadvisor.com | 0 of 3 | 0 of 3 | Blocked either way |
| Travel | kayak.com | 0 of 3 | 3 of 3 | Residential IP needed |
| Social, jobs and search | linkedin.com | 3 of 3 | 3 of 3 | No proxy needed |
| Social, jobs and search | indeed.com | 0 of 3 | 0 of 3 | Blocked either way |
| Social, jobs and search | google.com | 0 of 3 | 0 of 3 | Needs a browser |
“Blocked either way” means a plain HTTP client was blocked from both networks. A real browser or a scraping API that handles browser checks may well succeed on those sites; this test did not try.
What the results say
Public APIs, open data and documentation need no proxy. Every API and reference page we tried, except one, worked from a datacenter IP on every attempt. These services are built to be used by programs. What they want is a descriptive User-Agent, sensible request rates and, for some, an API key. A proxy adds cost and nothing else.
Many large sites do not block datacenter IPs at all. Three of the six news homepages, Amazon search, Airbnb search and a LinkedIn company page served real content to a cloud server three times out of three. For low-volume collection from sites like these, starting without a proxy is reasonable, as long as you stay within the site’s terms and rate limits.
Some sites block by IP reputation, and that is where residential exits matter. Walmart, Kayak and Booking.com refused the datacenter IP every time and served content through residential exits. These sites score the network a request comes from, and a cloud server’s address scores badly. This is the case residential proxies exist for.
Some sites block the client, not the IP. eBay, Etsy, Expedia, Tripadvisor, Indeed, Stack Overflow and three news sites blocked the plain Python client from both networks. They check how the client connects and behaves, such as its TLS fingerprint, its HTTP headers and whether it runs JavaScript, before they look at where it comes from. A residential IP does not fix a client that announces itself as a script. Our guide to TLS and HTTP/2 fingerprinting explains how those checks work.
Some pages need a browser whatever the network. Google Search returned a page asking for JavaScript to both networks, with no results in the HTML. For pages like that, the choice is between a headless browser and an API that returns the data directly.
How to tell what a site needs
You can sort most targets into these groups in an afternoon, before buying anything. The function below is the check we used in the test: it fetches a page and decides whether you got the real content, a block, an error or an empty shell.
import re
import requests
BLOCK = re.compile(r"captcha|access denied|are you a robot|just a moment|unusual traffic|"
r"verify you are human|request blocked|pardon our interruption|enable javascript and cookies",
re.IGNORECASE)
HEADERS = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
}
def check(url, marker, proxies=None):
"""Did this request get the real content, or a block, an error or an empty shell?"""
try:
response = requests.get(url, headers=HEADERS, proxies=proxies, timeout=30)
except requests.RequestException as error:
return {"url": url, "result": "error", "detail": type(error).__name__}
body = response.text
if response.status_code != 200:
result = "blocked" if response.status_code in (202, 401, 403, 429, 503) else "error"
elif BLOCK.search(body[:20000]) and marker.lower() not in body.lower():
result = "blocked"
elif marker.lower() not in body.lower():
result = "no content"
else:
result = "ok"
return {"url": url, "result": result, "status": response.status_code, "bytes": len(response.content)}Pick a marker that only appears in the content you want, such as a result card’s class name or a field name in a JSON response. Do not use a word from the page title: block pages and login walls often repeat it. Then run it on a few pages from each target:
from check import check
print(check("https://api.github.com/repos/python/cpython", '"full_name"'))
print(check("https://www.ebay.com/sch/i.html?_nkw=laptop", "srp-results")){'url': 'https://api.github.com/repos/python/cpython', 'result': 'ok', 'status': 200, 'bytes': 6617}
{'url': 'https://www.ebay.com/sch/i.html?_nkw=laptop', 'result': 'blocked', 'status': 403, 'bytes': 1832}Run it from where your scraper will actually run (a cloud server, not your laptop, if that is where production lives), a few times over a day. The results map to a decision:
| What you see from a cloud server | What it usually means | What to use |
|---|---|---|
| Real content, consistently | The site does not block datacenter traffic at your volume | No proxy, or a datacenter IP for scale |
| Blocks or challenges, and real content from a home connection | The site blocks by IP reputation | Residential proxies |
| Blocks from every network | The site checks the client | A real browser or a scraping API, usually with residential IPs underneath |
| Empty shells with no data | The content is rendered by JavaScript | A headless browser, or the API the page calls |
| Login walls | The content is not public | Do not collect it without permission |
Two refinements:
- Volume changes the answer. A site that ignores ten requests from a cloud server may block ten thousand. Re-test at the rate you plan to run, not just once.
- Location changes the answer. Sites serve different content and apply different rules by country. If you need data as seen from Germany, test from Germany.
The middle options
The choice is not only between no proxy and residential proxies.
- Datacenter IPs are cheap and fast, and fine for sites that do not block them. Our comparison of ISP and datacenter proxies covers when they are enough.
- ISP proxies are static addresses registered to consumer internet providers. They suit long sessions and logins to your own accounts, where you want the same IP for days. Shifter’s ISP proxies come with unlimited traffic, so they also suit heavy, steady workloads on sites that accept them.
- A scraping API handles the browser, retries and IP choice for you, and charges per successful page instead of per gigabyte. It is often the cheapest option for sites that check the client, because you do not have to build and maintain the browser layer yourself. See our guide to scraping JavaScript-heavy sites.
- Residential proxies are the right tool when a site judges the network a request comes from: when the same client succeeds from a home connection and fails from a cloud server.
FAQ
Do I need a proxy for web scraping?
Not always. In our test, 16 of 30 public pages, including every public API and most documentation sites, served real content to a cloud server with no proxy. Test your targets first, from where your scraper will run.
When do I need a residential proxy?
When a site blocks requests from datacenter IPs but serves the same client from a home connection. In our test, three of 30 sites (Walmart, Kayak and Booking.com) behaved that way.
Why am I still blocked with a residential proxy?
Usually because the site is checking the client, not the IP. Many sites block HTTP clients whose TLS fingerprint, headers or lack of JavaScript identify them as scripts. Eleven of the 30 pages in our test failed from both networks with a plain Python client. A real browser or a scraping API is the fix, not a different IP.
Are datacenter proxies enough for scraping?
For public APIs, documentation, many news sites and some large retailers, yes, at moderate volumes. For sites that score IP reputation, no. Test each target rather than assuming.
The bottom line
A residential proxy solves one specific problem: a site that judges requests by the network they come from. It does not help with sites that check the client, and it is unnecessary for public APIs and sites that do not block cloud traffic. Test each target from where your scraper will run, sort it into one of those groups, and pay for residential traffic only where it changes the result. Our web scraping best practices cover the rest of collecting responsibly.