Start Free TrialLog In

Glossary

What Is a User Agent?

A User-Agent is an HTTP request header that identifies the client software making the request — typically including browser name, version, OS, and rendering engine — like 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'.

Understand the User-Agent string, why it's the easiest signal websites use to identify scrapers, and how to rotate User-Agents alongside your IPs without blowing your fingerprint.

Explained

The User-Agent header is one of the oldest pieces of identification on the web. Every HTTP request a client sends includes a `User-Agent:` line that names the client software (browser or library), its version, the operating system, and often the rendering engine. Servers use it for content negotiation (mobile vs desktop layouts), analytics, and increasingly bot detection.

For scraping, the User-Agent is the simplest signal you can get wrong. Default User-Agents from common HTTP libraries (`python-requests/2.31.0`, `axios/1.5.0`, `okhttp/4.10.0`) are dead giveaways. Even Playwright and Puppeteer's default User-Agent contains `HeadlessChrome` which anti-bot systems instantly flag.

The naive fix is to set a Chrome UA on every request. The catch is that User-Agent alone isn't enough — modern fingerprinting cross-references the UA against `sec-ch-ua` client hints, the TLS fingerprint, the JavaScript navigator object, and the Accept-Language header. Setting a Chrome UA but leaving Python's TLS handshake exposed creates a bigger red flag than just leaving the default UA in place.

How It Works

When your client opens a connection to a server, the request line includes headers. The `User-Agent` line is one of them. Servers parse it (or just log it) and route the request based on what they see. A modern Chrome User-Agent on Windows looks like: `Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36`. The 'Mozilla/5.0' prefix is historical; every browser sends it for legacy compatibility reasons.

Real browsers also send `sec-ch-ua` ('Sec-CH-UA' Client Hints) headers in modern HTTP, which carry structured browser/version/platform data. Anti-bot systems compare those to the User-Agent string and flag inconsistencies. So User-Agent rotation has to be paired with consistent `sec-ch-ua-*` updates and matching TLS fingerprints.

Types

Desktop Browser User-Agents

Chrome, Firefox, Edge, Safari on Windows/macOS/Linux. The most common UA family for scraping general web targets.

Mobile Browser User-Agents

Chrome on Android, Safari on iOS, Samsung Internet. Required when scraping mobile-targeted sites or pairing with mobile proxies for fingerprint consistency.

Bot / Crawler User-Agents

Identifying UAs like `Googlebot/2.1`, `bingbot/2.0`. Used by legitimate search-engine crawlers; some sites whitelist these. Spoofing them risks legal and ToS issues.

Library Default User-Agents

`python-requests/X.Y`, `curl/X.Y`, `Mozilla/5.0 (compatible; Java/X.Y)`. Default UAs from HTTP libraries — never use these for production scraping; they're trivially identifiable.

Common Use Cases

Identifying client browser and OS for content negotiation
Server-side analytics and visitor segmentation
Bot detection (default library UAs are dead giveaways)
Mobile vs desktop layout selection
Caching by browser family
User-Agent rotation in scrapers to mimic varied real users
FAQ

Frequently asked FAQ questions

Common questions about user agent.

Use a current, mainstream User-Agent that matches a real browser version released within the last few months. Chrome on Windows or macOS is a safe default. Rotate across a small pool of current UAs rather than using one fixed UA for all requests.