Most advice about JavaScript-heavy sites stops at the first decision: does this page need a browser at all? That question matters, and it is answered in detail in when do you actually need a headless browser to scrape. A surprising share of “JavaScript sites” turn out not to need one.
This guide starts where that one ends. You have confirmed the target genuinely needs rendering. Now there is a second decision that shapes your cost and your on-call rota far more than the first: do you run the browsers yourself, or send the page to a web scraping API that runs them for you?
First, confirm the page really needs rendering
A quick check before committing to either path, because rendering is always the slower and heavier option.
Compare the source to the rendered page. Open the raw HTML response. If the data you want is in it, you do not need a browser.
Look for embedded state. Many JavaScript frameworks ship the initial page data as JSON inside a script tag, so the browser can hydrate the page without an extra request. If your data is in that JSON, a plain HTTP request plus a JSON parse is enough.
Watch the network requests. If the page fetches its data from a JSON endpoint after load, requesting that endpoint directly is usually cheaper than rendering the page around it.
If none of the three gets you the data, or the endpoint is signed, opaque or protected in ways you cannot reproduce, you need rendering. Read on.
What running your own browsers actually involves
A headless browser on a laptop is easy. A fleet of them in production is an operations problem, and the costs are easy to underestimate because none of them show up in a prototype.
Capacity. Each browser instance is memory-hungry, which caps how many can run on a machine and turns concurrency into an infrastructure bill.
Stability. Browsers hang, leak memory and crash on hostile pages. Production fleets need watchdogs, recycling and a queue that survives a worker dying mid-page.
Maintenance. Browser versions, automation libraries and the automation fingerprint all move. A fleet that worked last quarter can start failing without a line of your code changing.
Proxies. Rendered traffic still needs good IPs, wired into the browser correctly with authentication and per-context isolation. Getting this right is its own piece of work; see using residential proxies with Playwright.
Blocks and challenges. CAPTCHAs and anti-bot checks need detection, handling and retries, and a failed attempt still consumed the browser time it took.
Retries and waiting. Knowing when a dynamic page has finished loading, and what to do when it has not, is logic you write and maintain per site.
None of this is exotic. It is simply work that nobody bills a client for, and it grows with every new target.
What a web scraping API takes off your hands
A web scraping API turns that list into request parameters. With the Shifter Web Scraping API:
- Rendering.
render_js=1runs the page in headless Chrome, billed at the same one credit as a static fetch. - Waiting.
wait_for_cssholds the capture until a selector appears, andtimeoutcaps the browser time per page. - Interaction.
js_instructionsruns a chain ofscrollTo,clickandwaitsteps before capture, which covers cookie banners, load-more buttons and scroll-triggered content. - Structured output.
extract_rulesreturns JSON from CSS selectors, so there is no parser to deploy. - Retries. Failed fetches, CAPTCHAs and transient target errors are retried automatically, up to three times, with different proxies.
- Challenges. Stealth mode is on by default, and reCAPTCHA and hCaptcha are handled in-flight during rendered requests.
- IPs. Growth plans and above route through the residential and mobile pool with global geolocation. Starter uses datacenter IPs in the US and EU, which is enough for many unprotected pages.
- Sessions.
session_idkeeps cookies, browser state and the upstream IP across a multi-step flow, expiring after 10 minutes idle. - Billing. One credit per successful response. Failed requests, target errors and the API’s own retries are not charged.
Concurrency is capped per plan, from 20 on Starter to 500 on Enterprise, and requests above the cap return 429. The full parameter reference starts at rendering JavaScript.
When your own browsers are still the right call
An API is not always the answer, and it is worth being plain about where it is not.
Long interactive sessions. A flow that needs a browser to stay on a site for a long time, with pauses longer than a session’s idle window, fits your own browser better.
Arbitrary browser logic. If a page needs custom scripts, extensions or interaction beyond scrolling, clicking and waiting, a browser you control is more flexible.
Your own infrastructure is a requirement. Some workloads must run inside a specific network or environment for contractual or security reasons.
Very large, very stable volume. At sufficient scale on targets that rarely change, owned infrastructure can cost less per page, provided you already have the engineers to run it.
Testing and debugging. Visual regression, step-through debugging and anything where a human needs to watch the browser belong on your own machines.
A decision table per target
Make the choice per target, not per company. Most mature scraping stacks use all three paths.
| Target looks like | Best path |
|---|---|
| Data in the raw HTML or embedded JSON | Plain HTTP request |
| Data from a replayable JSON endpoint | Plain HTTP request to the endpoint |
| Rendered content, unprotected, low volume | Either; an API is less work |
| Rendered content behind anti-bot checks | Web scraping API |
| Rendered content across many markets | Web scraping API with per-request country |
| Long authenticated sessions or custom browser logic | Your own browsers with residential proxies |
| Huge, stable volume with an in-house team | Your own browsers, evaluated on total cost |
Compare on cost per usable page
The usual mistake in this decision is comparing an API’s per-request price against the cost of a server, and concluding the server is cheaper.
The fair comparison is cost per usable page. For your own fleet, that includes compute for browsers that sit idle or crash, proxy bandwidth for attempts that failed, and the engineering time spent on maintenance, retries and challenge handling, divided by the pages that actually produced correct data. For an API billed only on successful responses, failures are not charged, so the per-credit price is much closer to the true cost per usable page, though you still pay for successful pages that turn out to be useless, such as a changed selector returning empty fields.
Run the same sample of real target URLs through both paths for a week, count pages that returned correct, complete data, and compare the two costs per usable page. That number settles the argument faster than any feature list. Plans for the API are on the pricing page.
Where the two meet
The choice is not binary even for a single site. A common and sensible pattern is to use plain HTTP wherever a page does not need rendering, send rendered and protected pages to the API, and keep a small browser setup for the handful of flows that need custom logic.
Infinite scroll and load-more feeds sit right on this boundary, and the techniques for handling them inside a render are covered in how to handle infinite scroll and dynamic pagination. How an API assembles all of this behind one request is explained in how does a web scraping API work.
FAQ
Is rendering more expensive with an API?
In latency, yes, because a browser takes longer than a plain request. In credits, no: a rendered request costs the same one credit as a static fetch.
Can an API handle every anti-bot system?
Not every one, every time. Most checks are handled, and a consistently blocking target is something support can tune for. Measure your own success rate on your own targets before relying on it.
Do I still need proxies if I use an API?
No separate proxy setup. The API routes requests through its own pools, residential and mobile on Growth plans and above.
When should I move from an API to my own browsers?
When you need browser behaviour the API does not expose, or when a measured cost-per-usable-page comparison at your real volume favours owned infrastructure, including the engineering to run it.
The bottom line
Deciding that a site needs JavaScript rendering is the easy half. The expensive half is deciding who runs the browsers. Your own fleet buys flexibility and pays for it in capacity, stability, maintenance, proxies and challenge handling. An API trades that flexibility for parameters, retries and success-only billing.
Confirm a page truly needs rendering, choose per target rather than per company, and settle the build-versus-buy question on measured cost per usable page. The product is on the Web Scraping API page.