How to Scrape Pagination and Infinite Scroll Reliably at Scale

Pagination looks like the trivial part of a scrape. Increment page=2, loop until there is no next button, done. It is also where scrapers quietly lose data more often than anywhere else, and the loss is silent: you collect pages one through eight, miss nine through forty because the site capped you, and nothing errors. Or you double-count items because the list shifted under you. Or you stop early because loading paused and you mistook it for the end. Getting the last item, exactly once, and knowing you got it, is the entire game.

There are two shapes of the problem, classic pagination and infinite scroll, and they share one core question: how do I collect everything exactly once and know when I am actually finished? Here is how to answer it for both.

Classic pagination: know which kind you have

Before writing a loop, identify the pagination mechanism, because two of the three common kinds have traps that corrupt your data.

Offset or page-number pagination (?page=5 or ?offset=100&limit=25) is the most common and the most dangerous. Two things go wrong. First, drift: if the underlying list changes between requests, and on an active site it does, new items pushed to the top shift everything down, so page two now overlaps page one and an item slips between the cracks. Never dedup by position; dedup by a stable unique ID. Second, deep-pagination caps: many sites refuse to serve past page 100 or offset 10,000, so the tail is simply unreachable by offset. When you hit that wall, you cannot page your way to the end, you have to partition the data another way, by date range, category, or price band, and paginate within each slice so no slice exceeds the cap.

Cursor or keyset pagination (?after=<token>) is the robust kind. The response hands you a cursor for the next batch, you follow it, and you stop when it is absent. It is immune to drift because the cursor points at a stable position in the data, not an offset that shifts. Prefer it whenever the site offers it, and never try to construct or guess a cursor, treat it as opaque and only follow the one you were given.

cursor, seen = None, set()
while True:
    resp = fetch(url, params={"after": cursor} if cursor else {})
    for item in resp["items"]:
        if item["id"] not in seen:      # dedup by stable id, never by position
            seen.add(item["id"]); yield item
    cursor = resp.get("next_cursor")
    if not cursor:                       # absent cursor is the real end signal
        break

The single most useful move here is to look at the network layer, not the rendered HTML. Open the target in a browser, watch the XHR/fetch requests, and you will very often find a clean JSON endpoint with cursor pagination sitting behind the page. Calling that directly is faster, lighter on bandwidth, and far more reliable than scraping page-by-page HTML.

Infinite scroll: it is usually cursor pagination in disguise

Infinite scroll almost never needs a real browser. Under the hood, scrolling just triggers the same paginated fetch, and if you find that underlying request in the network tab, you can call it directly with its cursor exactly like an API and skip the browser entirely, which is dramatically cheaper on bandwidth and time.

When the site genuinely requires rendering, drive it with Playwright or Puppeteer, and respect three traps that catch everyone.

First, know what triggers a load: scroll position, an IntersectionObserver sentinel near the bottom, or a “load more” button. Trigger the right one.

Second, wait for the new batch to actually arrive, not for a fixed timer. A sleep(2) is a race condition: sometimes the content has loaded, sometimes it has not, and on a slow proxy hop it often has not. Wait until the item count increases or the network goes idle.

prev = 0
while True:
    page.mouse.wheel(0, 20000)                     # trigger the next batch
    page.wait_for_function(                         # wait for real arrival, not a timer
        "n => document.querySelectorAll('.item').length > n", arg=prev)
    items = page.query_selector_all('.item')
    if len(items) == prev:                          # no growth after a real wait
        break                                       # ...but see termination below
    prev = len(items)

Third, and the one that silently truncates the most data: virtualized lists. Libraries like react-window remove off-screen rows from the DOM to stay fast, so if you scroll to the bottom and only then scrape the DOM, you get the last visible window and nothing else. You have to extract items as they appear during the scroll, not once at the end.

Knowing when you are actually done

Termination is the hardest part, because “loading stopped” has two very different causes that look identical: you reached the genuine end, or the site rate-limited you mid-sequence and quietly stopped serving more. Treating the second as the first is exactly how you ship a dataset that is missing its tail.

Three defenses. Retry a stalled load a couple of times before declaring the end, so a slow batch is not mistaken for completion. Compare against a total when the site exposes one, a “1,240 results” header is a checksum: if you collected 900, you were truncated, not finished. And treat a load that stops right after a burst of requests as a suspected soft block rather than the end, and retry that tail with a fresh identity instead of accepting partial data. This is the same silent-failure problem the pipeline monitoring fill-rate and expected-count checks are built to catch across a whole run.

Dedup and completeness, always

Two habits make the difference between a complete dataset and a plausible-looking incomplete one. Deduplicate by a stable unique ID, never by page number, order, or content hash of a whole row, because ordering shifts and rows get lightly edited. And track collected-versus-expected counts wherever the site gives you a total, so truncation shows up as a number that does not add up rather than a gap nobody notices until much later.

Where proxies fit

A paginated sequence is usually one logical session, and it should look like one. Use a sticky session so every page of a single query exits through the same IP: rotating mid-sequence can trip anti-bot systems that expect one visitor to page coherently, and on personalized or geo-varying sites it can even return inconsistent results between pages. Give each query its own sticky session and rotate between queries, not within one.

Deep pagination also means many requests to a single host in a short window, which is precisely where you get rate-limited and blocked. Pace the sequence, back off on 429s rather than hammering, and lean on a clean pool so you are challenged less often to begin with. Because a mid-sequence block masquerades as the end, better IP reputation does double duty here: it both reduces the truncation and, when combined with expected-count checks, makes the truncation you do hit visible instead of silent.

The bottom line

Pagination is not the trivial part, it is where completeness is won or lost. Identify the real mechanism and prefer cursor pagination or the underlying JSON endpoint over offset-and-HTML. Dedup by stable ID, never by position. For infinite scroll, call the underlying fetch when you can, and when you must render, wait for real loads rather than timers and extract items as they appear so virtualized lists do not eat your data. Above all, treat “loading stopped” as suspicious until an expected-count check or a retried tail proves it was really the end, and run each paginated query on its own sticky session so the sequence stays coherent.

Do that and you collect the whole list, exactly once, and know that you did. Point the sequence at a clean residential gateway so the tail does not turn into a wall of blocks, and the per-GB pricing lets you crawl deep pagination without a per-request meter working against you.