The best way to scrape an infinite scroll feed is usually not to scroll at all. Behind most feeds sits a paginated request, often cursor-based, and replaying that request directly is faster, cheaper and more reliable than driving a browser. That approach is covered in detail in how to scrape pagination and infinite scroll reliably.
This guide is for the cases where that does not work. The request is signed with a short-lived token. The response is an opaque blob the page decodes client-side. The feed only loads when an element enters the viewport. The “next page” is a button wired to state you cannot reconstruct. In those cases you have to handle the scrolling and clicking inside a real render, and there are a handful of ways it goes wrong.
The examples use the Shifter Web Scraping API, which runs the page in headless Chrome and accepts a chain of browser actions before capture.
Four kinds of dynamic pagination
Identify which one you have before writing anything, because each needs a different technique.
| Pattern | What triggers the next batch | Technique |
|---|---|---|
| Scroll-triggered feed | An element near the bottom entering the viewport | Scroll to a sentinel, wait, repeat |
| Load-more button | A click on a button that appends items | Click, wait for new items, repeat |
| URL-state pagination | The page updates the URL as you move through results | Fetch those URLs directly, no scrolling |
| Virtualised list | Scrolling, but old rows are removed from the DOM as new ones appear | Capture in windows, or use the underlying request |
The third is worth checking first because it is the cheapest to handle. Scroll a feed in a normal browser and watch the address bar. If a page or offset parameter changes, the site has already given you a paginated URL, and each page is an ordinary request.
The fourth is the one that silently loses data, and it gets its own section below.
Rendering and waiting correctly
Every dynamic technique starts with the same two controls. render_js=1 runs the page in headless Chrome, at the same one credit per successful request as a static fetch. wait_for_css holds the capture until a selector exists in the DOM, so you never extract from a page that has not populated yet. The rendering controls are documented in rendering JavaScript.
Choose the wait selector carefully. Waiting for the list container is not enough, because many frameworks render an empty container first and fill it later. Wait for the first item in the list instead.
Scroll-triggered feeds: scroll to a sentinel
Browser actions go in js_instructions, a JSON array of steps executed in order before capture. The documented actions are scrollTo, click and wait.
The pattern for a scroll-triggered feed is to scroll to an element that sits after the list, wait for the next batch to load, and repeat. Because the list grows, that element moves further down each time, so scrolling to it again triggers the next load.
import json
import os
import requests
API = "https://scrape.shifter.io/v1"
instructions = [
{"action": "click", "selector": "button.accept-cookies"},
{"action": "scrollTo", "selector": "footer", "timeout": 5000, "block": "start"},
{"action": "wait", "duration": 2000},
{"action": "scrollTo", "selector": "footer", "timeout": 5000, "block": "start"},
{"action": "wait", "duration": 2000},
{"action": "scrollTo", "selector": "footer", "timeout": 5000, "block": "start"},
{"action": "wait", "duration": 2000},
]
rules = {
"items": {
"selector": "article.card",
"type": "list",
"item": {
"link": {"selector": "a.card-link", "output": "@href"},
"name": {"selector": "h3", "output": "text"},
"price": {"selector": ".price", "output": "text"},
},
}
}
params = {
"api_key": os.environ["SHIFTER_API_KEY"],
"url": "https://shop.example.com/category/shoes",
"render_js": 1,
"wait_for_css": "article.card",
"js_instructions": json.dumps(instructions),
"extract_rules": json.dumps(rules),
}
resp = requests.get(API, params=params, timeout=120)
resp.raise_for_status()
items = resp.json()["items"]A few details in there are deliberate.
The cookie banner is dismissed first, because an overlay can intercept the scroll or the click. The wait between scrolls gives the network request time to complete and the DOM time to update; too short and you capture before the batch arrives. extract_rules with a list type returns every card as a JSON object, so there is no parser on your side, and requests URL-encodes both JSON parameters for you. The extraction syntax is in extraction rules.
Test the chain on a sample page before running it at scale, and tune the number of scroll steps and the wait duration against how quickly that site actually loads.
Load-more buttons: click, then wait for growth
A load-more button is the same loop with a different trigger:
[
{"action": "click", "selector": "button.load-more", "timeout": 3000},
{"action": "wait", "duration": 2000},
{"action": "click", "selector": "button.load-more", "timeout": 3000},
{"action": "wait", "duration": 2000}
]Two things catch people out. The button selector often changes state while loading, gaining a disabled class or a spinner, so the second click can fire before the button is clickable again unless the wait is long enough. And the button usually disappears when there is nothing left to load, which is a useful completeness signal but means your chain needs to tolerate the last clicks having nothing to act on. Again, test against the real site.
The time budget of a single render
Everything above happens inside one browser session with a time limit. wait_for_css times out after 30 seconds by default, and timeout caps how long the browser may spend on the page. A feed with thousands of items will not finish loading inside one render, however many scroll steps you chain.
So split the problem rather than lengthening the chain.
Narrow the result set. Filters, sort orders and category facets usually produce smaller feeds. Twenty narrow feeds that each load fully are more reliable than one enormous feed that never finishes.
Page through filtered URLs. Many sites combine a filter with URL state, which turns an unbounded scroll into a finite set of ordinary requests.
Fall back to the underlying request when the feed is genuinely long and cannot be narrowed. Fetch it without rendering; for endpoints that return JSON, auto_parser=1 returns the parsed body.
Virtualised lists: the silent data loss
Some feeds, particularly very long ones, use list virtualisation. Only the rows near the viewport exist in the DOM. As you scroll down, rows at the top are removed.
Scroll a virtualised list to the bottom and capture, and you get the last screen of items, not every item you scrolled past. Nothing errors. The extraction returns a clean, plausible, incomplete list.
Detect it before trusting any output. Run the scroll chain, then check whether the items from the first screen are still present in the captured result. If they are gone, the list is virtualised, and scrolling-then-capturing cannot work. Use URL-state pagination or the underlying request instead.
Dynamic pagination across requests
When each page is a separate request that depends on server-side state, such as a cursor stored against your session, hold that state constant across the walk with session_id. A session persists cookies, browser state and the upstream IP across requests, and expires after 10 minutes idle. Keep the country the same for the life of a session, since switching mid-walk can invalidate cookies tied to locale. The details are in sessions and proxies.
params.update({
"session_id": "shoes-walk-07",
"country": "de",
})Keep a walk moving. A session that sits idle while your code does something slow between pages will expire and break the cursor.
Knowing you got everything
A scraper that stops early looks exactly like a scraper that finished. Build completeness checks into every run.
- Compare against the displayed total. Many feeds show a result count. If the page says 1,284 and you extracted 960, you stopped early.
- Deduplicate on a stable key, such as the item’s link or ID, never on position. Scroll feeds routinely re-render items, and position shifts as content is inserted.
- Check for the end marker. A disappeared load-more button or an end-of-results message is positive confirmation. Its absence after your last step means the chain was too short.
- Track completeness over time. A category that yielded 1,200 items yesterday and 400 today has usually changed its markup or its loading behaviour, not its inventory.
Credits and latency: choosing the approach per site
| Approach | Credits | Latency | Reliability risk |
|---|---|---|---|
| Replay the underlying request | None if sent directly; one per page through the API | Low | Signed or opaque requests |
| URL-state pagination | One per page | Low to moderate | Needs a paginated URL |
| Scroll or click chain in one render | One for the whole chain | High | Time budget, virtualisation |
A render that scrolls through several batches costs one credit, where replaying the same batches as separate requests costs one credit each. The trade is latency and the time budget: long chains are slower and more likely to time out. Use scroll chains for moderate feeds where the underlying request is impractical, and reach for the other two for everything else.
FAQ
Should I always render JavaScript for infinite scroll?
No. Check for URL-state pagination and for a replayable underlying request first. Rendering is the fallback, not the default.
How many scroll steps should the chain have?
As many as the site needs to load the batches you want within the time budget. Measure on a sample page, and if you need more than the budget allows, narrow the feed instead.
Why does my extraction return fewer items than I scrolled past?
Almost always list virtualisation, where rows leave the DOM as you scroll. Check whether first-screen items survive to the capture.
Does each scroll step cost a credit?
No. The whole chain runs inside one request, and one successful request is one credit.
The bottom line
Infinite scroll is cursor pagination with a browser in front of it, and when you can reach the cursor directly, you should. When you cannot, handle it inside the render: wait for the first item rather than the container, scroll to a sentinel or click load-more with enough wait between steps, respect the time budget by narrowing the feed, test for virtualisation before trusting a capture, and verify completeness against the site’s own totals.
For deciding when a rendered API request is the right tool in the first place, see when you need a web scraping API for JavaScript-heavy sites. The product is on the web scraping API with JS rendering page, with plans on the pricing page.