A common pattern in support conversations: someone switches to residential proxies, the block rate improves, and then a stubborn subset of targets keeps refusing them anyway. The IP is clean, the geography is right, and the requests still get challenged. Almost always the answer is in the headers, because the address gets you to the door and the headers decide whether what walks through looks like a browser.
The mistake underneath most header problems is thinking of it as a list of values to set correctly. It is not. It is a set of claims that have to agree with each other and with everything else about your connection, and a request built from mismatched parts is more suspicious than one with no effort at all.
Consistency beats any individual value
Start here, because it reframes everything that follows. A server does not evaluate your User-Agent in isolation. It sees a bundle: the UA string, the accompanying client hints, the header set and its order, the Accept-Language value, the TLS handshake underneath, and the IP the whole thing arrived from. Real browsers produce internally consistent bundles because the same software generated all of it.
Scrapers produce inconsistent ones by accident. A Chrome UA arriving with the header set of a Python HTTP library, a Windows UA over a TLS fingerprint that belongs to a Linux tool, a US exit sending Accept-Language: de-DE, or a claimed browser that never requests the assets a browser would. None of those individually says “bot”, but the contradiction does, and contradictions are much easier to detect reliably than any single signal. This is the same logic as the device-and-network pairing in antidetect browsers: every layer has to tell the same story.
So the objective is not the most convincing UA string. It is a request where the UA, the headers, the TLS fingerprint, and the exit address all describe the same plausible visitor.
What a real browser actually sends
If you only set User-Agent, you are already inconsistent, because no browser sends a UA and nothing else. A modern Chrome request for a page carries, at minimum, a set along these lines:
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br, zstd
sec-ch-ua: "Chromium";v="141", "Not?A_Brand";v="24", "Google Chrome";v="141"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1Three things in there are worth understanding rather than copying.
Client hints must match the UA. The sec-ch-ua brand list, sec-ch-ua-platform, and sec-ch-ua-mobile are a structured restatement of what the UA string already claims. If your UA says Chrome on Windows and your platform hint says macOS, or the version in the brand list disagrees with the version in the UA, you have contradicted yourself in two adjacent headers. Sending no client hints at all while claiming to be a recent Chrome is also a mismatch, since a real Chrome sends them.
Sec-Fetch headers describe context. They tell the server what kind of request this is: a top-level navigation, a subresource fetch, a same-origin XHR. A page load is Dest: document, Mode: navigate, Site: none when opened directly or same-origin when following an internal link. An XHR to an API is Dest: empty, Mode: cors. Getting these wrong is a giveaway precisely because they are automatic in a browser and easy to forget in a script.
Accept-Encoding is a claim you must honour. Advertise br and zstd only if your client can actually decompress them. Some libraries advertise encodings they then fail to handle, which produces either errors or a fallback that differs from what a browser would negotiate.
Match Accept-Language to your exit country
This one is specific to proxy work and is the most common self-inflicted mismatch. If you exit through a German residential IP and send Accept-Language: en-US, you have described a visitor whose browser is configured for American English sitting on a German home connection. That happens in real life, but it is unusual enough to be a signal, and more practically it can change what you get back: many sites serve content based on that header, so a geo-targeted collection job can retrieve the wrong language while appearing to work.
Tie the language to the exit, ideally in the same place you choose the country, so the two can never drift apart:
import requests
MARKETS = {
"us": "en-US,en;q=0.9",
"de": "de-DE,de;q=0.9,en;q=0.8",
"fr": "fr-FR,fr;q=0.9,en;q=0.8",
"br": "pt-BR,pt;q=0.9,en;q=0.8",
}
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36")
def fetch(url, country):
proxy = f"http://customer-USERNAME-country-{country}:PASSWORD@p.shifter.io:443"
headers = {
"User-Agent": UA,
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,"
"image/avif,image/webp,*/*;q=0.8",
"Accept-Language": MARKETS[country], # follows the exit, always
"Accept-Encoding": "gzip, deflate, br",
"sec-ch-ua": '"Chromium";v="141", "Not?A_Brand";v="24", '
'"Google Chrome";v="141"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"', # agrees with the UA
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Upgrade-Insecure-Requests": "1",
}
return requests.get(url, headers=headers,
proxies={"http": proxy, "https": proxy}, timeout=20)The same discipline applies to timezone and locale when you are driving a browser, where those are separately observable and should match the exit as well, which gets tighter when you are working at city-level targeting.
Do not rotate the User-Agent randomly
This is advice that circulates widely and does more harm than good. Picking a random UA per request produces a pattern no real population has: one IP that is Chrome on Windows, then Safari on a Mac, then Firefox on Linux, within a minute. Worse, if you hold a sticky session so one address serves a multi-step flow, changing the UA mid-flow means the same visitor apparently swapped devices between clicking search and viewing a result.
The coherent model is one identity per session. Pick a plausible UA, keep it for the life of that session, and keep everything else consistent with it. If you want variety across your fleet, vary between sessions rather than between requests, and vary in realistic proportions rather than uniformly across every browser that has ever existed. Keep the versions current too: a UA claiming a browser version from three years ago is itself anomalous, since real installations update.
Headers are only one layer
Worth being honest about the ceiling here. Perfect headers do not make a Python client look like Chrome, because the layer underneath still differs. The TLS handshake and HTTP/2 settings your client produces form a fingerprint of their own, and a Chrome UA over a fingerprint that says Python is exactly the contradiction discussed at the start. That mismatch is the subject of TLS and HTTP/2 fingerprinting, and it is why some heavily defended targets stay out of reach for plain HTTP clients regardless of header work, which is one of the cases for reaching for a headless browser.
Header order matters for the same reason. Browsers emit headers in a stable order; many HTTP libraries emit them alphabetically or in insertion order, which is another way the bundle can betray its origin even when every value is right. Some clients let you control ordering, and where it matters, matching a real browser’s order is worth the effort. The broader catalogue of these signals is in the fingerprints that block data extraction, and the common self-inflicted versions are in mistakes that trigger detection.
A short checklist
Send a full browser header set, not just a User-Agent. Make client hints agree with the UA on brand, version, platform, and mobile flag. Set Sec-Fetch headers to describe the actual request type. Tie Accept-Language to the exit country and keep them in the same code path. Advertise only the encodings you can decode. Hold one identity per session rather than rotating per request, and keep UA versions current. Then check whether your TLS fingerprint agrees with the browser you are claiming to be, because that is the layer headers cannot fix.
If a target still refuses you after all of that, the problem has moved elsewhere: pacing, IP reputation, or behavioural signals, which are the wider set covered in avoiding blocks and scraping heavily protected sites.
The bottom line
Headers are the second half of the identity your proxy starts. A clean residential address makes the connection unremarkable; a coherent header bundle makes the request unremarkable, and coherence is the whole game. Every claim has to agree with every other claim: client hints with the UA, language with the exit country, fetch metadata with the request type, encodings with your actual capabilities, and the whole set with the TLS fingerprint underneath. One identity per session, held steady, beats clever rotation every time.
That connection layer is what residential proxies provide, real home-grade addresses with country and city targeting so the geography your headers claim is the geography you are actually exiting from, billed per GB so tuning requests to fetch less costs you less.