Start Free TrialLog In

Glossary

What Is a User Agent?

A User-Agent is an HTTP request header that identifies the client software making the request — typically including browser name, version, OS, and rendering engine — like 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'.

Understand the User-Agent string, why it's the easiest signal websites use to identify scrapers, and how to rotate User-Agents alongside your IPs without blowing your fingerprint.

Explained

The User-Agent header is one of the oldest pieces of identification on the web. Every HTTP request a client sends includes a `User-Agent:` line that names the client software (browser or library), its version, the operating system, and often the rendering engine. Servers use it for content negotiation (mobile vs desktop layouts), analytics, and increasingly bot detection.

For scraping, the User-Agent is the simplest signal you can get wrong. Default User-Agents from common HTTP libraries (`python-requests/2.31.0`, `axios/1.5.0`, `okhttp/4.10.0`) are dead giveaways. Even Playwright and Puppeteer's default User-Agent contains `HeadlessChrome` which anti-bot systems instantly flag.

The naive fix is to set a Chrome UA on every request. The catch is that User-Agent alone isn't enough — modern fingerprinting cross-references the UA against `sec-ch-ua` client hints, the TLS fingerprint, the JavaScript navigator object, and the Accept-Language header. Setting a Chrome UA but leaving Python's TLS handshake exposed creates a bigger red flag than just leaving the default UA in place.

How It Works

When your client opens a connection to a server, the request line includes headers. The `User-Agent` line is one of them. Servers parse it (or just log it) and route the request based on what they see. A modern Chrome User-Agent on Windows looks like: `Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36`. The 'Mozilla/5.0' prefix is historical; every browser sends it for legacy compatibility reasons.

Real browsers also send `sec-ch-ua` ('Sec-CH-UA' Client Hints) headers in modern HTTP, which carry structured browser/version/platform data. Anti-bot systems compare those to the User-Agent string and flag inconsistencies. So User-Agent rotation has to be paired with consistent `sec-ch-ua-*` updates and matching TLS fingerprints.

Types

Desktop Browser User-Agents

Chrome, Firefox, Edge, Safari on Windows/macOS/Linux. The most common UA family for scraping general web targets.

Mobile Browser User-Agents

Chrome on Android, Safari on iOS, Samsung Internet. Required when scraping mobile-targeted sites or pairing with mobile proxies for fingerprint consistency.

Bot / Crawler User-Agents

Identifying UAs like `Googlebot/2.1`, `bingbot/2.0`. Used by legitimate search-engine crawlers; some sites whitelist these. Spoofing them risks legal and ToS issues.

Library Default User-Agents

`python-requests/X.Y`, `curl/X.Y`, `Mozilla/5.0 (compatible; Java/X.Y)`. Default UAs from HTTP libraries — never use these for production scraping; they're trivially identifiable.

Common Use Cases

Identifying client browser and OS for content negotiation
Server-side analytics and visitor segmentation
Bot detection (default library UAs are dead giveaways)
Mobile vs desktop layout selection
Caching by browser family
User-Agent rotation in scrapers to mimic varied real users
FAQ

Frequently asked questions

Common questions about user agent.

Use a current, mainstream User-Agent that matches a real browser version released within the last few months. Chrome on Windows or macOS is a safe default. Rotate across a small pool of current UAs rather than using one fixed UA for all requests.

Rotate across requests, but not arbitrarily. Stick with one UA for the lifetime of a session (a cluster of related requests for a workflow), then switch when the session ends. Random per-request UA changes are themselves a fingerprint signal because real users don't change browsers between page loads.

Most likely a fingerprint mismatch. If you set a Chrome UA but your TLS handshake is Python's default, anti-bot vendors see the inconsistency. Pair the UA with matching `sec-ch-ua` headers, matching TLS fingerprint (use `curl_cffi` or a real browser), and a believable Accept-Language. Then UA rotation actually helps.

Keep it current. Browser versions release every 2-6 weeks. A User-Agent with Chrome 95 in 2026 is suspicious because no real user is on Chrome 95 anymore. Update your pool monthly or pull from a current-UA source like ua-parser/ua-parser-js public datasets.

Yes if your User-Agent claims to be Chromium-based (Chrome, Edge, Opera). Modern Chromium browsers send `sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform` headers. If your UA is Chrome but you're missing these, anti-bot systems flag the mismatch.

Yes, in most jurisdictions. The User-Agent header is just a string the client controls. Setting it to anything is legal in itself; legality of what you do with the resulting access depends on the destination's ToS, anti-bot laws (CFAA in the US), and what data you collect. Public-data scraping with a spoofed UA is broadly legal; bypassing authentication is not.