Residential proxies are what we sell, so it may seem odd for us to write about when you do not need one. But teams that buy proxies for every job overspend, and teams that blame the proxy for every failure waste weeks. Knowing which sites need a residential IP, which need nothing, and which need something else entirely is the cheapest optimisation in web data collection.
To make that concrete, we ran a small test. On 9 October 2026 we requested 30 public pages, three times each, from a cloud server with a datacenter IP address, then again through Shifter residential proxies with US exits. Both runs used the same simple Python HTTP client. Sixteen of the 30 pages worked from the datacenter IP every time. Three worked only through residential exits. The remaining eleven failed both ways, for reasons a residential IP does not fix.
Our test: 30 pages, datacenter IP vs residential proxies
The targets covered six kinds of site: public APIs and open data, reference and documentation sites, news homepages, e-commerce search pages, travel search pages, and a set of social, jobs and search pages. Each request used the same desktop browser User-Agent and was checked for real content, not just a status code: a response only counted as a success if it contained a marker that appears only in the actual content, such as a search result card. Block pages that return status 200, which are common, therefore counted as failures.
We chose the markers by inspecting saved responses from both networks before the final run, and dropped five candidate pages where we could not tell real content from a login wall or an empty JavaScript shell.
| Category | Site | Datacenter IP | Residential exits | Verdict |
|---|---|---|---|---|
| Public APIs and open data | en.wikipedia.org | 3 of 3 | 3 of 3 | No proxy needed |
| Public APIs and open data | api.github.com | 3 of 3 | 3 of 3 | No proxy needed |
| Public APIs and open data | data.sec.gov | 3 of 3 | 3 of 3 | No proxy needed |
| Public APIs and open data | api.worldbank.org | 3 of 3 | 3 of 3 | No proxy needed |
| Public APIs and open data | ecb.europa.eu | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | docs.python.org | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | developer.mozilla.org | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | rfc-editor.org | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | en.wikipedia.org | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | arxiv.org | 3 of 3 | 3 of 3 | No proxy needed |
| Reference and documentation | stackoverflow.com | 0 of 3 | 0 of 3 | Blocked either way |
| News | bbc.com | 3 of 3 | 3 of 3 | No proxy needed |
| News | reuters.com | 0 of 3 | 0 of 3 | Blocked either way |
| News | nytimes.com | 0 of 3 | 0 of 3 | Blocked either way |
| News | theguardian.com | 3 of 3 | 3 of 3 | No proxy needed |
| News | edition.cnn.com | 3 of 3 | 3 of 3 | No proxy needed |
| News | bloomberg.com | 0 of 3 | 0 of 3 | Blocked either way |
| E-commerce search | amazon.com | 3 of 3 | 2 of 3 | No proxy needed |
| E-commerce search | walmart.com | 0 of 3 | 3 of 3 | Residential IP needed |
| E-commerce search | ebay.com | 0 of 3 | 0 of 3 | Blocked either way |
| E-commerce search | bestbuy.com | 0 of 3 | 0 of 3 | Timed out either way |
| E-commerce search | etsy.com | 0 of 3 | 0 of 3 | Blocked either way |
| Travel | booking.com | 0 of 3 | 2 of 3 | Residential IP needed |
| Travel | expedia.com | 0 of 3 | 0 of 3 | Blocked either way |
| Travel | airbnb.com | 3 of 3 | 2 of 3 | No proxy needed |
| Travel | tripadvisor.com | 0 of 3 | 0 of 3 | Blocked either way |
| Travel | kayak.com | 0 of 3 | 3 of 3 | Residential IP needed |
| Social, jobs and search | linkedin.com | 3 of 3 | 3 of 3 | No proxy needed |
| Social, jobs and search | indeed.com | 0 of 3 | 0 of 3 | Blocked either way |
| Social, jobs and search | google.com | 0 of 3 | 0 of 3 | Needs a browser |
“Blocked either way” means a plain HTTP client was blocked from both networks. A real browser or a scraping API that handles browser checks may well succeed on those sites; this test did not try.
By category, the results add up like this:
| Category | Pages | No proxy needed | Residential IP needed | Failed both ways |
|---|---|---|---|---|
| Public APIs and open data | 5 | 5 | 0 | 0 |
| Reference and documentation | 6 | 5 | 0 | 1 |
| News | 6 | 3 | 0 | 3 |
| E-commerce search | 5 | 1 | 1 | 3 |
| Travel | 5 | 1 | 2 | 2 |
| Social, jobs and search | 3 | 1 | 0 | 2 |
| Total | 30 | 16 | 3 | 11 |
What the 30-page test shows
Public APIs, open data and documentation need no proxy. Every API and reference page we tried, except one, worked from a datacenter IP on every attempt. These services are built to be used by programs. What they want is a descriptive User-Agent, sensible request rates and, for some, an API key. A proxy adds cost and nothing else.
Many large sites do not block datacenter IPs at all. Three of the six news homepages, Amazon search, Airbnb search and a LinkedIn company page served real content to a cloud server three times out of three. For low-volume collection from sites like these, starting without a proxy is reasonable, as long as you stay within the site’s terms and rate limits.
Amazon and Airbnb each worked three times out of three from the datacenter IP and two out of three through residential exits. With three attempts per site, one miss is within normal variation, not evidence that residential IPs do worse there. Each residential request leaves from a different home connection, and an individual exit can be slow, or recently seen by the same site, in a way a fixed cloud server is not. The lesson is to test several times before deciding, which the check below makes easy.
Some sites block by IP reputation, and that is where residential exits matter. Walmart, Kayak and Booking.com refused the datacenter IP every time and served content through residential exits. These sites score the network a request comes from, and a cloud server’s address scores badly. This is the case residential proxies exist for.
Some sites block the client, not the IP. eBay, Etsy, Expedia, Tripadvisor, Indeed, Stack Overflow and three news sites blocked the plain Python client from both networks. They check how the client connects and behaves, such as its TLS fingerprint, its HTTP headers and whether it runs JavaScript, before they look at where it comes from. A residential IP does not fix a client that announces itself as a script. Our guide to TLS and HTTP/2 fingerprinting explains how those checks work.
Some pages need a browser whatever the network. Google Search returned a page asking for JavaScript to both networks, with no results in the HTML. For pages like that, the choice is between a headless browser and an API that returns the data directly.
How to tell what a site needs
You can sort most targets into these groups in an afternoon, before buying anything. The function below is the check we used in the test: it fetches a page and decides whether you got the real content, a block, an error or an empty shell.
import re
import requests
BLOCK = re.compile(r"captcha|access denied|are you a robot|just a moment|unusual traffic|"
r"verify you are human|request blocked|pardon our interruption|enable javascript and cookies",
re.IGNORECASE)
HEADERS = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
}
def check(url, marker, proxies=None):
"""Did this request get the real content, or a block, an error or an empty shell?"""
try:
response = requests.get(url, headers=HEADERS, proxies=proxies, timeout=30)
except requests.RequestException as error:
return {"url": url, "result": "error", "detail": type(error).__name__}
body = response.text
if response.status_code != 200:
result = "blocked" if response.status_code in (202, 401, 403, 429, 503) else "error"
elif BLOCK.search(body[:20000]) and marker.lower() not in body.lower():
result = "blocked"
elif marker.lower() not in body.lower():
result = "no content"
else:
result = "ok"
return {"url": url, "result": result, "status": response.status_code, "bytes": len(response.content)}
Pick a marker that only appears in the content you want, such as a result card’s class name or a field name in a JSON response. Do not use a word from the page title: block pages and login walls often repeat it. Then run it on a few pages from each target:
from check import check
print(check("https://api.github.com/repos/python/cpython", '"full_name"'))
print(check("https://www.ebay.com/sch/i.html?_nkw=laptop", "srp-results"))
{'url': 'https://api.github.com/repos/python/cpython', 'result': 'ok', 'status': 200, 'bytes': 6617}
{'url': 'https://www.ebay.com/sch/i.html?_nkw=laptop', 'result': 'blocked', 'status': 403, 'bytes': 1832}
Run it from where your scraper will actually run (a cloud server, not your laptop, if that is where production lives), a few times over a day. The results map to a decision:
| What you see from a cloud server | What it usually means | What to use |
|---|---|---|
| Real content, consistently | The site does not block datacenter traffic at your volume | No proxy, or a datacenter IP for scale |
| Blocks or challenges, and real content from a home connection | The site blocks by IP reputation | Residential proxies |
| Blocks from every network | The site checks the client | A real browser or a scraping API, usually with residential IPs underneath |
| Empty shells with no data | The content is rendered by JavaScript | A headless browser, or the API the page calls |
| Login walls | The content is not public | Do not collect it without permission |
Two refinements:
- Volume changes the answer. A site that ignores ten requests from a cloud server may block ten thousand. Re-test at the rate you plan to run, not just once.
- Location changes the answer. Sites serve different content and apply different rules by country. If you need data as seen from Germany, test from Germany.
When location-specific data needs a residential proxy
Our test asked one question: does the site serve its content at all? A second question often matters more: does it serve the content a real person in a given place would see?
A cloud server sits in a handful of data-centre locations, and many sites recognise data-centre addresses and show them a default version: the global homepage, prices in US dollars, no local stock, or no ads at all. If your job depends on what users in a specific country or city see, you need an exit there, on a network those users actually use. That covers:
- Local search results and rankings, which change by city.
- Prices, availability and delivery options, which retailers and travel sites set by market.
- Ads and landing pages, which advertisers target by location and which often do not show to data-centre traffic. Checking them is the core of ad verification.
- Content that is licensed or restricted by country.
In these cases the site may never block a datacenter IP, and you would still collect the wrong data without a residential exit in the right place.
The middle options
The choice is not only between no proxy and residential proxies.
- Datacenter IPs are cheap and fast, and fine for sites that do not block them. Our comparison of ISP and datacenter proxies covers when they are enough.
- ISP proxies are addresses registered to consumer internet providers, kept for you over a longer period. They suit long sessions and logins to your own accounts. Shifter’s ISP proxies stay in the city and on the ISP (ASN) you choose for the whole billing period and come with unlimited traffic, so they also suit heavy, steady workloads on sites that accept them.
- A scraping API handles the browser, retries and IP choice for you, and charges per successful page instead of per gigabyte. It is often the cheapest option for sites that check the client, because you do not have to build and maintain the browser layer yourself. See our guide to scraping JavaScript-heavy sites.
- Residential proxies are the right tool when a site judges the network a request comes from: when the same client succeeds from a home connection and fails from a cloud server.
FAQ
Do I need a proxy for web scraping?
Not always. In our test, 16 of 30 public pages, including every public API and most documentation sites, served real content to a cloud server with no proxy. Test your targets first, from where your scraper will run.
When do I need a residential proxy?
When a site blocks requests from datacenter IPs but serves the same client from a home connection. In our test, three of 30 sites (Walmart, Kayak and Booking.com) behaved that way.
Why am I still blocked with a residential proxy?
Usually because the site is checking the client, not the IP. Many sites block HTTP clients whose TLS fingerprint, headers or lack of JavaScript identify them as scripts. Eleven of the 30 pages in our test failed from both networks with a plain Python client. A real browser or a scraping API is the fix, not a different IP.
Are datacenter proxies enough for scraping?
For public APIs, documentation, many news sites and some large retailers, yes, at moderate volumes. For sites that score IP reputation, no. Test each target rather than assuming.
What is the difference between a datacenter IP and a residential IP?
A datacenter IP belongs to a hosting or cloud provider. A residential IP belongs to a consumer internet provider and is used by a real home connection. Many sites trust residential addresses more, because ordinary visitors use them.
How do I test whether a site blocks datacenter IPs?
Request a few pages from a cloud server and check the response for a marker that only appears in the real content, not just the status code. Then repeat through a residential exit. If the cloud server is refused and the residential exit is served, the site blocks by IP reputation.
Do I need a residential proxy for location-specific data?
Usually, yes. Even sites that never block cloud servers often show them a default version of the page. To see the local results, prices or ads that users in a country or city see, you need an exit in that place.
Why did Amazon and Airbnb work less often through residential proxies in your test?
Each failed once in three residential attempts, which is within normal variation for three tries. Every residential request leaves from a different home connection, and a single exit can be slow or recently seen by the site. Test several times before drawing a conclusion.
The bottom line
A residential proxy solves one specific problem: a site that judges requests by the network they come from. It does not help with sites that check the client, and it is unnecessary for public APIs and sites that do not block cloud traffic. Test each target from where your scraper will run, sort it into one of those groups, and pay for residential traffic only where it changes the result. Our web scraping best practices cover the rest of collecting responsibly.