The teams that defend against bots and the teams that build them have been locked in a slow-motion arms race for about fifteen years. The frontier moved every few years, and every shift forced a different kind of investment from data teams. It’s worth knowing the rough timeline because the current state of play isn’t obvious from the outside, and a lot of advice on the internet is two anti-bot generations out of date.
Generation 1: IP block lists and rate limits
Early defenses were procedural. A request came in, the server noted the IP, and if that IP did anything obviously machine-like, too many requests per minute, too predictable a sequence, too small a User-Agent, it got rate-limited or banned outright. Lists of known datacenter IP ranges floated around. Anything from AWS or DigitalOcean was treated with suspicion.
This was the era when the answer to “how do I scrape X” was almost always “use a proxy.” Specifically, a datacenter proxy. Buy a /24, rotate through it, problem solved.
It worked because the defenders didn’t have much else.
Generation 2: CAPTCHAs and the user-friction tax
When IP-level defenses started failing, defenders pushed friction onto the user. CAPTCHAs became universal, first the image-matching ones, then reCAPTCHA, then invisible reCAPTCHA, then hCaptcha and Turnstile and Arkose.
CAPTCHAs work, in the sense that they break naive scrapers. They also work in the sense that they annoy real users badly enough to measurably reduce conversion. Most sites that used aggressive CAPTCHA layers in the 2017–2020 window have since pulled back, because the friction cost outweighed the bot cost.
From the scraper side, CAPTCHAs spawned an entire industry of human-in-the-loop solver services. Pay a few cents per solve, queue your CAPTCHAs to a panel of workers in low-cost geographies, get the token back, continue scraping. Not elegant, but it worked.
Generation 3: Browser and behavioral fingerprinting
The third generation moved up the stack. Sites started fingerprinting the client itself, not just the IP, but the browser. Canvas fingerprinting, WebGL signatures, font lists, audio context, timezone, language preferences, screen dimensions, available plugins, the order in which TLS ciphers were proposed in the handshake (JA3 / JA4 fingerprints), the timing of mouse movements, the cadence of keystrokes.
Headless Chromium without specific hardening leaks dozens of these signals. Defenders built libraries (Fingerprint.js, the various commercial equivalents) that scored requests across this surface and rejected anything that looked too clean, too mechanical, or too unlike a real user’s browser.
This is when scraping got hard. A residential IP was no longer enough on its own. You needed to drive a real browser, or a sufficiently good headless one with patched fingerprints, and you needed to make its behavior look natural. The proxy was still essential, but it had become one component in a stack rather than the whole answer.
Generation 4: Network-level reputation
The current frontier is the most expensive one for everyone. Defenders now buy reputation data from network providers like Cloudflare, Akamai, and a dozen smaller specialists. Every IP gets a reputation score that aggregates signals from millions of sites, has this IP been seen logging in to bank accounts, doing checkout flows, opening regular Gmail? Or has it been seen hitting login endpoints in patterns that suggest credential stuffing, posting comments at machine cadences, scraping competitor pricing pages at 3 AM Tuesday?
A datacenter IP has, almost by definition, a thin or negative network history. It hasn’t done normal human things. A residential IP, by contrast, has years of mundane traffic behind it, the household it serves uses it for Netflix, Steam, Zoom, normal browsing. That reputation is what protects scrapers today.
ISP proxies sit in an interesting middle: they’re issued by real residential ISPs (so the upstream provider matches a household IP block) but they’re allocated to data centers and held statically. They’re harder to detect than pure datacenter IPs but easier than rotating residential, and they’re priced accordingly.
What this means for data teams in 2026
A few things have stayed true through all four generations and will keep being true:
The right proxy is necessary but not sufficient. A real residential IP gets you past the network-level gate. From there your client, your headers, your TLS fingerprint, and your behavioral patterns still need to be plausible.
Block rate is the only metric that matters. Not pool size, not country count, not advertised features. If your scrapes return clean HTML at acceptable cost, the infrastructure is working. If they don’t, no amount of marketing claims will help.
The right proxy depends on the workflow. Per-request rotation across a giant residential pool is correct for fan-out price-monitoring jobs. Sticky sessions on a single residential IP for ten minutes is correct for multi-page flows. Pinned ISP IPs are correct for account management and any workflow that needs the same identity over weeks or months. Don’t pick one tool for all jobs.
Anti-bot will keep moving up the stack. The next layer is probably more aggressive behavioral analysis at the edge, machine-learned classifiers that look at multi-minute session shapes, not single-request features. This is already in production at the largest sites. The defenders’ arms race compounds, and so does the cost of staying in it.
The honest takeaway: nobody who tells you scraping is “solved” is correct. It’s an operations problem, and the operations are what you pay for. Residential proxies remain the foundation because they’re still the only place where the upstream IP carries the right kind of history. Everything else sits on top of that.