More unique IPs than any network on the planet, in the world's largest residential pool. That's Shifter.More unique IPs than any network, in the world's largest pool. That's Shifter.

See the numbers

When You Don't Need a Residential Proxy: 30 Pages Tested

We fetched 30 public pages from a datacenter IP and through residential exits: 16 needed no proxy, 3 needed a residential IP. How to tell which kind you have.

Product Marketing · Updated October 29, 2026 · 14 min read
16311no proxy neededresidentialblocked anyway
30 public sites. Only the blue three needed a residential IP to get through.
Residential Proxies
On this page
  1. Our test: 30 pages, datacenter IP vs residential proxies
  2. What the 30-page test shows
  3. How to tell what a site needs
  4. When location-specific data needs a residential proxy
  5. The middle options
  6. FAQ
  7. The bottom line

Residential proxies are what we sell, so it may seem odd for us to write about when you do not need one. But teams that buy proxies for every job overspend, and teams that blame the proxy for every failure waste weeks. Knowing which sites need a residential IP, which need nothing, and which need something else entirely is the cheapest optimisation in web data collection.

To make that concrete, we ran a small test. On 9 October 2026 we requested 30 public pages, three times each, from a cloud server with a datacenter IP address, then again through Shifter residential proxies with US exits. Both runs used the same simple Python HTTP client. Sixteen of the 30 pages worked from the datacenter IP every time. Three worked only through residential exits. The remaining eleven failed both ways, for reasons a residential IP does not fix.

Our test: 30 pages, datacenter IP vs residential proxies

The targets covered six kinds of site: public APIs and open data, reference and documentation sites, news homepages, e-commerce search pages, travel search pages, and a set of social, jobs and search pages. Each request used the same desktop browser User-Agent and was checked for real content, not just a status code: a response only counted as a success if it contained a marker that appears only in the actual content, such as a search result card. Block pages that return status 200, which are common, therefore counted as failures.

We chose the markers by inspecting saved responses from both networks before the final run, and dropped five candidate pages where we could not tell real content from a login wall or an empty JavaScript shell.

CategorySiteDatacenter IPResidential exitsVerdict
Public APIs and open dataen.wikipedia.org3 of 33 of 3No proxy needed
Public APIs and open dataapi.github.com3 of 33 of 3No proxy needed
Public APIs and open datadata.sec.gov3 of 33 of 3No proxy needed
Public APIs and open dataapi.worldbank.org3 of 33 of 3No proxy needed
Public APIs and open dataecb.europa.eu3 of 33 of 3No proxy needed
Reference and documentationdocs.python.org3 of 33 of 3No proxy needed
Reference and documentationdeveloper.mozilla.org3 of 33 of 3No proxy needed
Reference and documentationrfc-editor.org3 of 33 of 3No proxy needed
Reference and documentationen.wikipedia.org3 of 33 of 3No proxy needed
Reference and documentationarxiv.org3 of 33 of 3No proxy needed
Reference and documentationstackoverflow.com0 of 30 of 3Blocked either way
Newsbbc.com3 of 33 of 3No proxy needed
Newsreuters.com0 of 30 of 3Blocked either way
Newsnytimes.com0 of 30 of 3Blocked either way
Newstheguardian.com3 of 33 of 3No proxy needed
Newsedition.cnn.com3 of 33 of 3No proxy needed
Newsbloomberg.com0 of 30 of 3Blocked either way
E-commerce searchamazon.com3 of 32 of 3No proxy needed
E-commerce searchwalmart.com0 of 33 of 3Residential IP needed
E-commerce searchebay.com0 of 30 of 3Blocked either way
E-commerce searchbestbuy.com0 of 30 of 3Timed out either way
E-commerce searchetsy.com0 of 30 of 3Blocked either way
Travelbooking.com0 of 32 of 3Residential IP needed
Travelexpedia.com0 of 30 of 3Blocked either way
Travelairbnb.com3 of 32 of 3No proxy needed
Traveltripadvisor.com0 of 30 of 3Blocked either way
Travelkayak.com0 of 33 of 3Residential IP needed
Social, jobs and searchlinkedin.com3 of 33 of 3No proxy needed
Social, jobs and searchindeed.com0 of 30 of 3Blocked either way
Social, jobs and searchgoogle.com0 of 30 of 3Needs a browser

“Blocked either way” means a plain HTTP client was blocked from both networks. A real browser or a scraping API that handles browser checks may well succeed on those sites; this test did not try.

By category, the results add up like this:

CategoryPagesNo proxy neededResidential IP neededFailed both ways
Public APIs and open data5500
Reference and documentation6501
News6303
E-commerce search5113
Travel5122
Social, jobs and search3102
Total3016311

What the 30-page test shows

Public APIs, open data and documentation need no proxy. Every API and reference page we tried, except one, worked from a datacenter IP on every attempt. These services are built to be used by programs. What they want is a descriptive User-Agent, sensible request rates and, for some, an API key. A proxy adds cost and nothing else.

Many large sites do not block datacenter IPs at all. Three of the six news homepages, Amazon search, Airbnb search and a LinkedIn company page served real content to a cloud server three times out of three. For low-volume collection from sites like these, starting without a proxy is reasonable, as long as you stay within the site’s terms and rate limits.

Amazon and Airbnb each worked three times out of three from the datacenter IP and two out of three through residential exits. With three attempts per site, one miss is within normal variation, not evidence that residential IPs do worse there. Each residential request leaves from a different home connection, and an individual exit can be slow, or recently seen by the same site, in a way a fixed cloud server is not. The lesson is to test several times before deciding, which the check below makes easy.

Some sites block by IP reputation, and that is where residential exits matter. Walmart, Kayak and Booking.com refused the datacenter IP every time and served content through residential exits. These sites score the network a request comes from, and a cloud server’s address scores badly. This is the case residential proxies exist for.

Some sites block the client, not the IP. eBay, Etsy, Expedia, Tripadvisor, Indeed, Stack Overflow and three news sites blocked the plain Python client from both networks. They check how the client connects and behaves, such as its TLS fingerprint, its HTTP headers and whether it runs JavaScript, before they look at where it comes from. A residential IP does not fix a client that announces itself as a script. Our guide to TLS and HTTP/2 fingerprinting explains how those checks work.

Some pages need a browser whatever the network. Google Search returned a page asking for JavaScript to both networks, with no results in the HTML. For pages like that, the choice is between a headless browser and an API that returns the data directly.

How to tell what a site needs

You can sort most targets into these groups in an afternoon, before buying anything. The function below is the check we used in the test: it fetches a page and decides whether you got the real content, a block, an error or an empty shell.

import re

import requests

BLOCK = re.compile(r"captcha|access denied|are you a robot|just a moment|unusual traffic|"
                   r"verify you are human|request blocked|pardon our interruption|enable javascript and cookies",
                   re.IGNORECASE)
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}


def check(url, marker, proxies=None):
    """Did this request get the real content, or a block, an error or an empty shell?"""
    try:
        response = requests.get(url, headers=HEADERS, proxies=proxies, timeout=30)
    except requests.RequestException as error:
        return {"url": url, "result": "error", "detail": type(error).__name__}
    body = response.text
    if response.status_code != 200:
        result = "blocked" if response.status_code in (202, 401, 403, 429, 503) else "error"
    elif BLOCK.search(body[:20000]) and marker.lower() not in body.lower():
        result = "blocked"
    elif marker.lower() not in body.lower():
        result = "no content"
    else:
        result = "ok"
    return {"url": url, "result": result, "status": response.status_code, "bytes": len(response.content)}

Pick a marker that only appears in the content you want, such as a result card’s class name or a field name in a JSON response. Do not use a word from the page title: block pages and login walls often repeat it. Then run it on a few pages from each target:

from check import check

print(check("https://api.github.com/repos/python/cpython", '"full_name"'))
print(check("https://www.ebay.com/sch/i.html?_nkw=laptop", "srp-results"))
{'url': 'https://api.github.com/repos/python/cpython', 'result': 'ok', 'status': 200, 'bytes': 6617}
{'url': 'https://www.ebay.com/sch/i.html?_nkw=laptop', 'result': 'blocked', 'status': 403, 'bytes': 1832}

Run it from where your scraper will actually run (a cloud server, not your laptop, if that is where production lives), a few times over a day. The results map to a decision:

What you see from a cloud serverWhat it usually meansWhat to use
Real content, consistentlyThe site does not block datacenter traffic at your volumeNo proxy, or a datacenter IP for scale
Blocks or challenges, and real content from a home connectionThe site blocks by IP reputationResidential proxies
Blocks from every networkThe site checks the clientA real browser or a scraping API, usually with residential IPs underneath
Empty shells with no dataThe content is rendered by JavaScriptA headless browser, or the API the page calls
Login wallsThe content is not publicDo not collect it without permission

Two refinements:

  • Volume changes the answer. A site that ignores ten requests from a cloud server may block ten thousand. Re-test at the rate you plan to run, not just once.
  • Location changes the answer. Sites serve different content and apply different rules by country. If you need data as seen from Germany, test from Germany.

When location-specific data needs a residential proxy

Our test asked one question: does the site serve its content at all? A second question often matters more: does it serve the content a real person in a given place would see?

A cloud server sits in a handful of data-centre locations, and many sites recognise data-centre addresses and show them a default version: the global homepage, prices in US dollars, no local stock, or no ads at all. If your job depends on what users in a specific country or city see, you need an exit there, on a network those users actually use. That covers:

  • Local search results and rankings, which change by city.
  • Prices, availability and delivery options, which retailers and travel sites set by market.
  • Ads and landing pages, which advertisers target by location and which often do not show to data-centre traffic. Checking them is the core of ad verification.
  • Content that is licensed or restricted by country.

In these cases the site may never block a datacenter IP, and you would still collect the wrong data without a residential exit in the right place.

The middle options

The choice is not only between no proxy and residential proxies.

  • Datacenter IPs are cheap and fast, and fine for sites that do not block them. Our comparison of ISP and datacenter proxies covers when they are enough.
  • ISP proxies are addresses registered to consumer internet providers, kept for you over a longer period. They suit long sessions and logins to your own accounts. Shifter’s ISP proxies stay in the city and on the ISP (ASN) you choose for the whole billing period and come with unlimited traffic, so they also suit heavy, steady workloads on sites that accept them.
  • A scraping API handles the browser, retries and IP choice for you, and charges per successful page instead of per gigabyte. It is often the cheapest option for sites that check the client, because you do not have to build and maintain the browser layer yourself. See our guide to scraping JavaScript-heavy sites.
  • Residential proxies are the right tool when a site judges the network a request comes from: when the same client succeeds from a home connection and fails from a cloud server.

FAQ

Do I need a proxy for web scraping?

Not always. In our test, 16 of 30 public pages, including every public API and most documentation sites, served real content to a cloud server with no proxy. Test your targets first, from where your scraper will run.

When do I need a residential proxy?

When a site blocks requests from datacenter IPs but serves the same client from a home connection. In our test, three of 30 sites (Walmart, Kayak and Booking.com) behaved that way.

Why am I still blocked with a residential proxy?

Usually because the site is checking the client, not the IP. Many sites block HTTP clients whose TLS fingerprint, headers or lack of JavaScript identify them as scripts. Eleven of the 30 pages in our test failed from both networks with a plain Python client. A real browser or a scraping API is the fix, not a different IP.

Are datacenter proxies enough for scraping?

For public APIs, documentation, many news sites and some large retailers, yes, at moderate volumes. For sites that score IP reputation, no. Test each target rather than assuming.

What is the difference between a datacenter IP and a residential IP?

A datacenter IP belongs to a hosting or cloud provider. A residential IP belongs to a consumer internet provider and is used by a real home connection. Many sites trust residential addresses more, because ordinary visitors use them.

How do I test whether a site blocks datacenter IPs?

Request a few pages from a cloud server and check the response for a marker that only appears in the real content, not just the status code. Then repeat through a residential exit. If the cloud server is refused and the residential exit is served, the site blocks by IP reputation.

Do I need a residential proxy for location-specific data?

Usually, yes. Even sites that never block cloud servers often show them a default version of the page. To see the local results, prices or ads that users in a country or city see, you need an exit in that place.

Why did Amazon and Airbnb work less often through residential proxies in your test?

Each failed once in three residential attempts, which is within normal variation for three tries. Every residential request leaves from a different home connection, and a single exit can be slow or recently seen by the site. Test several times before drawing a conclusion.

The bottom line

A residential proxy solves one specific problem: a site that judges requests by the network they come from. It does not help with sites that check the client, and it is unnecessary for public APIs and sites that do not block cloud traffic. Test each target from where your scraper will run, sort it into one of those groups, and pay for residential traffic only where it changes the result. Our web scraping best practices cover the rest of collecting responsibly.

Ready to get started?

Try Shifter's residential proxies, 205M+ IPs, 195+ countries, from $0.10/GB.

Get Started