Scraping

The Cost of a Stale User-Agent: Measuring Browser Version Drift

Scrapers often hardcode a browser version and forget it. We requested 397 top sites claiming Chrome versions up to 6.5 years old. Refusals rose every year.

Chris Collins

October 5, 2026 · 8 min read

Every scraper that sets a User-Agent header picks a browser version, and almost none of them ever change it. The string goes into a config file the day the project starts, and the project keeps announcing itself as that browser for years. Meanwhile real browsers move on: Chrome shipped 13 major versions between the end of September 2025 and the end of September 2026.

Does the drift matter? Anti-bot systems know which browser versions real visitors use, so a version nobody has run for years stands out. We measured how much, by requesting the homepages of 397 top sites while claiming five different Chrome versions, from the current release to one six and a half years old.

Key takeaways

  • Older claimed versions were refused more often, every time. 17.1% of homepages challenged or blocked the current Chrome version; 19.9% refused a one-year-old version and 22.7% a six-and-a-half-year-old one.
  • The effect only went one way. 22 sites that served the current version refused the oldest one; no site refused the current version and served an old one.
  • Two identical requests for the current version produced identical outcomes on all 397 sites, so the differences are not noise.
  • Sites behind Akamai reacted first: 9 of the 11 sites that refused a version just one year old were Akamai-protected. Cloudflare-protected sites mostly reacted only to the oldest version.
  • Keep the version current, keep it consistent with the rest of the request, and check it on a schedule, as you would any dependency.

How we measured

We took 400 sites at random from the homepages in our earlier anti-bot stack lookup of the top 1,000 domains in the Tranco ranking. On 5 October 2026 we requested each homepage six times, a second apart and in a random order per site, with a standard Windows Chrome User-Agent string claiming these versions:

Claimed versionStable releaseAge at test
Chrome 154, requested twice22 September 2026Current
Chrome 14130 September 2025About 1 year
Chrome 12917 September 2024About 2 years
Chrome 1107 February 2023About 3.5 years
Chrome 804 February 2020About 6.5 years

The repeated request for the current version is the control: if a site varies on its own, it shows up as a difference between those two. All requests came from the same Python HTTP client and the same connection; only the version number in the User-Agent changed. Three sites failed to respond to at least one request and were excluded, leaving 397.

What we found

Claimed versionServedChallengedBlockedChallenged or blocked
Chrome 154, first request329422617.1%
Chrome 154, repeat329422617.1%
Chrome 141318433619.9%
Chrome 129315433920.7%
Chrome 110314434020.9%
Chrome 80307523822.7%

The two requests for the current version agreed on every single site, which makes the rest of the table easy to read: every difference is the version number.

Counting only sites that served the current version both times, the number that refused the older version grew steadily with its age:

Claimed versionSites that served the current version but refused this one
Chrome 141, 1 year old11
Chrome 129, 2 years old14
Chrome 110, 3.5 years old15
Chrome 80, 6.5 years old22

No site did the opposite. A stale version never helped.

Who reacts, and how

Cross-referencing the sites with the protection detected in the earlier lookup shows that vendors draw the line in different places:

  • A year was enough for Akamai. Of the 11 sites that refused Chrome 141, 9 were Akamai-protected; all 11 answered with HTTP 403, and 10 of those were plain blocks rather than challenges.
  • Cloudflare reacted to the very old version. Of the 22 sites that refused Chrome 80, 10 were Cloudflare-protected and returned a challenge; 9 more were Akamai.
  • Pages rarely told the visitor. Only two sites showed an “update your browser” style message to Chrome 80 that they did not show to the current version. Most simply refused.

This is consistent with how bot management works. A real visitor on a one-year-old browser is uncommon, and on a six-year-old one, rare, so an old version is a weak signal on its own and a stronger one combined with anything else unusual about the request.

The code

The function below runs the same comparison against any URL: it requests the page claiming each version, in random order, and returns the status, outcome and size for each, plus whether the page tells the visitor their browser is outdated. List a version twice to get a noise control.

import random
import re
import time

import requests

CHROME = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{v}.0.0.0 Safari/537.36"
OUTDATED = re.compile(r"(browser (is|you are using is) (not supported|out of date|outdated|no longer supported)"
                      r"|update your browser|upgrade your browser|unsupported browser|outdated browser)", re.I)


def classify(response):
    """served, challenged or blocked, from the status and well-known challenge markers."""
    body = response.text[:20000].lower()
    if (response.headers.get("cf-mitigated", "").lower() == "challenge"
            or response.headers.get("x-amzn-waf-action", "").lower() == "challenge"
            or "captcha-delivery.com" in body):
        return "challenged"
    if response.status_code in (401, 403, 405, 429) or response.status_code >= 500:
        return "blocked"
    return "served"


def ua_drift_check(url, versions, pause=1.0, seed=None):
    """Request the same URL claiming each Chrome version, in random order, and compare what comes back.
    A version may be listed twice, as a control for how much the site varies on its own."""
    order = list(enumerate(versions))
    random.Random(seed).shuffle(order)
    results = [None] * len(versions)
    for i, v in order:
        headers = {"User-Agent": CHROME.format(v=v), "Accept": "text/html,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.9"}
        try:
            r = requests.get(url, headers=headers, timeout=20)
            results[i] = {"version": v, "status": r.status_code, "outcome": classify(r), "bytes": len(r.content),
                          "outdated_notice": bool(OUTDATED.search(r.text))}
        except requests.RequestException as e:
            results[i] = {"version": v, "error": type(e).__name__}
        time.sleep(pause)
    return results

Run it against the sites you depend on before and after a User-Agent change, and keep the results: if a target starts refusing the version you send, you will know whether the version is the reason.

Limits of the measurement

  • Homepages, one network, one moment. Deeper pages, other countries and other times may behave differently.
  • Not a browser. Our client sent a Chrome User-Agent from a Python HTTP library, so every request already had a network fingerprint that did not match Chrome, as explained in TLS and HTTP/2 fingerprinting. The comparison isolates the effect of the version number; it does not show how a real browser of each version would fare.
  • The baseline is not zero. 17% of homepages refused even the current version from this client, which is a reminder that a correct User-Agent is necessary, not sufficient.

We should also own a finding about ourselves. Our anti-bot lookup earlier in the week used a Chrome 129 User-Agent: a two-year-old string, copied from an older script. By this measurement, it cost a few percentage points of homepages.

Keeping your User-Agent current

  • Treat it as a dependency. Put the browser version in one place, and update it on a schedule; Chrome releases a new major version roughly every four weeks.
  • Stay within a few versions of current. In our data, a version one year old was already refused more often than the current one.
  • Keep everything consistent. A current Chrome version with an old header order, missing client hints or a non-browser network fingerprint is still inconsistent; setting the right headers and User-Agent covers the full set.
  • Use one identity per session. Rotating versions within a session is more suspicious than any single version.
  • Test in CI. A scheduled check of key targets with the current and previous configuration catches drift before it shows up in your data; running scrapers in CI/CD shows how to set that up.
  • Watch the refusal rate. A slow rise in 403s and challenges is the typical symptom of drift; it belongs in a target health score.

The bottom line

A browser version is not a setting you choose once. In our measurement of 397 top sites, every year of drift in the claimed Chrome version added refusals, the effect never went the other way, and some protection services reacted to a version only a year out of date.

Keep the version current, keep the rest of the request consistent with it, and measure the effect on the targets that matter to you. It is one of the cheapest reliability improvements a scraping project can make.

Sources and references

  • Chromium Dash, release schedule, for stable release dates and the current version.
  • Tranco, list Q2K34.
  • Requests made by Shifter on 5 October 2026 using the code above.

Ready to get started?

Try Shifter's residential proxies, 205M+ IPs, 195+ countries, from $0.10/GB.

Get Started