You have done everything right. A clean residential IP, a real Chrome User-Agent, believable headers, sensible rate limits. And a heavily protected target still blocks you instantly, sometimes before it sends a single byte of the actual page. It is tempting to blame the IP and rotate, but rotating does not help, and that is the tell. You are being fingerprinted at a layer most scrapers never think about: the HTTP client itself, at the TLS handshake and the HTTP/2 connection, before your request is even read.
This is the third layer of identity, separate from your IP and separate from the browser and device fingerprint an antidetect browser manages. Understanding it is what separates a scraper that works on easy sites from one that works on the hard ones.
Three layers, not one
When a site decides whether you are a real user, it looks at three independent things:
- The network layer: your IP address and its reputation. A datacenter IP or a flagged one fails here.
- The client layer: how your HTTP client opens the connection, its TLS handshake and HTTP/2 behavior. A scripting library fails here even from a perfect IP.
- The device layer: the browser fingerprint, canvas, fonts,
navigator, and so on. An automated browser can fail here.
A clean residential IP solves the first layer and nothing else. If your TLS handshake announces “I am Python,” no IP in the world makes you look human. This is why the hardest targets block you before the page loads: they reject the connection at the client layer, and the IP never gets a chance to matter.
TLS fingerprinting, or JA3
Every HTTPS connection starts with a TLS handshake, and the very first message, the ClientHello, is surprisingly revealing. In it, the client lists the exact cipher suites it supports, in a specific order, along with its TLS extensions, supported elliptic curves, signature algorithms, and more. That combination is characteristic of the software making the connection, and it is remarkably stable.
Chrome produces one distinctive ClientHello. Firefox produces another. Python’s requests, which uses the system OpenSSL, produces one that looks nothing like either. Go’s net/http produces its own. Anti-bot vendors hash these values into a compact fingerprint, the well-known formats are JA3 and its successor JA4, and maintain a picture of what real browsers look like. When your ClientHello hashes to something that is obviously a scripting library and not a browser, you are flagged at the handshake, before you have sent a request line, a header, or a cookie.
The cruel part is the contradiction it exposes. You set User-Agent: Chrome, but your TLS fingerprint says OpenSSL-via-Python. A real Chrome would never produce that combination, so the mismatch is not just a failed disguise, it is a positive signal that something is pretending. You are worse off than if you had sent no User-Agent at all.
HTTP/2 fingerprinting
Say your TLS fingerprint is convincing. There is another layer right behind it. Real browsers speak HTTP/2, and how they speak it is also a fingerprint. When an HTTP/2 connection opens, the client sends a SETTINGS frame with particular values (header table size, max concurrent streams, initial window size), advertises a specific window update, and orders its pseudo-headers (:method, :authority, :scheme, :path) in a characteristic way. Browsers even send stream-priority information in a recognizable pattern.
HTTP client libraries get these details wrong in telltale ways, or fall back to HTTP/1.1 entirely, which on a modern site is itself suspicious. So a scraper can pass the TLS check and still be caught one layer up, because its HTTP/2 SETTINGS and header ordering do not match any real browser. And on plain HTTP/1.1, the order and casing of your headers is one more fingerprint: browsers send headers in a consistent order that most libraries do not reproduce.
Why setting headers does not fix it
The reason this is so persistent is that none of it lives in the values you can set. You can spoof a User-Agent, add an Accept-Language, copy a browser’s header list exactly, and still be caught, because the fingerprint is a property of how the connection is made, not what you put in the request. The cipher order in your ClientHello, the HTTP/2 SETTINGS your library emits, the sequence in which headers hit the wire, these are decided by your TLS and HTTP stack, not by your code. You cannot header your way out of a JA3 mismatch.
How to look like a real browser at the network layer
There are two honest ways to present a browser-grade client fingerprint.
Use a real browser. Playwright, Puppeteer, or Selenium drive real Chromium, so their TLS and HTTP/2 fingerprints are, by definition, a real browser’s. For the most aggressively protected targets this is the most robust route, at the cost of running a browser per worker.
Use a TLS-impersonating HTTP client. When you want to stay lightweight, a class of clients exists specifically to mimic a browser’s ClientHello and HTTP/2 settings while remaining a simple HTTP call. Tools like curl-impersonate and its Python binding curl_cffi, tls-client, and Go’s utls let you send a request that fingerprints as Chrome or Firefox without launching one.
from curl_cffi import requests
# Presents a real Chrome TLS + HTTP/2 fingerprint, not python-requests'.r = requests.get( "https://example.com", impersonate="chrome", proxies={"https": "http://customer-USER-country-us:PASS@p.shifter.io:443"},)print(r.status_code)That single impersonate argument is the difference between a ClientHello that says Chrome and one that says OpenSSL. Paired with a clean residential IP, it clears both the network and the client layer at once, which is what the hard targets are checking.
Keep every layer consistent
The through-line is consistency. Your story has to match across all three layers: a Chrome User-Agent needs a Chrome TLS fingerprint, Chrome HTTP/2 settings, and Chrome header order, exiting through an IP whose geography matches the locale you claim. Anti-bot systems increasingly score the agreement between these signals, not each one in isolation, so a single contradiction, a browser UA on a library handshake, a US persona on a foreign IP, is a louder signal than any one layer being slightly off. This is the same coherence principle behind the mistakes that trigger detection: every layer has to tell the same story.
Diagnosing a client-fingerprint block
You can usually tell a client-layer block apart from an IP-layer block by how it behaves. If you are blocked immediately, on a site known for aggressive protection, and rotating IPs changes nothing but switching to a real browser suddenly works, that is a client-fingerprint block, not an IP problem. Conversely, if a plain HTTP client works fine most of the time and only some IPs get challenged, that points back at IP reputation. Knowing which layer failed tells you which lever to pull, instead of rotating IPs against a problem that has nothing to do with the IP. This is the same “read the failure correctly” discipline as diagnosing why requests time out and detecting a silent block.
The bottom line
Heavily protected sites judge you on three layers, and a clean IP only answers one of them. The client layer, your TLS handshake and HTTP/2 behavior, gets scripting libraries blocked before their requests are even read, and no amount of header spoofing fixes it, because the fingerprint is in how the connection is made. To scrape the hard targets with a high success rate, present a browser-grade client fingerprint, either by driving a real browser or by using a TLS-impersonating client, and keep it consistent with your headers and your IP’s geography.
A clean residential IP remains necessary, it is the layer that gets you past reputation, but on the toughest sites it is not sufficient on its own. Pair it with a real browser fingerprint at the network layer and every signal lined up, and the block that used to hit before the page even loaded stops happening. The per-GB pricing lets you test that combination against your own hardest targets without a per-request meter fighting you.