How to Use a Web Scraping API: Beginner-Friendly Walkthrough

A web scraping API does the hard parts of collecting web pages for you: it picks the proxy, runs the browser when a page needs one, retries failures, and hands back the HTML or the data you asked for. You send one request with a URL; the service deals with everything between you and the site.

This walkthrough takes you from your first request to a small, reliable scraper, using Shifter’s Web Scraping API as the example. The steps and patterns are the same for most services of this kind. If you want the background on what happens behind the request first, our explainer on how a web scraping API works covers it.

Step 1: get an API key

Every request is authenticated with an API key. In Shifter’s panel, open Web Scraping API, then API Keys, and copy an active key.

Keep the key out of your code. Store it in an environment variable, so it never ends up in a repository or a screenshot:

export SHIFTER_SCRAPER_KEY="your-key-here"

Step 2: make your first request

The API is a single endpoint, https://scrape.shifter.io/v1. Pass your key and the page you want as query parameters:

curl "https://scrape.shifter.io/v1?api_key=$SHIFTER_SCRAPER_KEY&url=https://example.com"

The response is the page’s HTML, fetched through the API’s proxies. That is all a basic request needs.

Step 3: turn on JavaScript rendering when the page needs it

Many modern sites send a nearly empty page and build the content in the browser with JavaScript. If the HTML you get back is missing the data you can see in your browser, add render_js=1. The API then loads the page in a headless browser and returns the rendered result. On Shifter, a rendered request costs the same as a plain one.

Two related settings help with slow pages:

• wait_for_css waits until a specific element appears before returning, so you do not get the page before its data has loaded.
• timeout gives slow sites more time.

Rendering is slower than a plain fetch, so use it only for pages that need it. Our guide to scraping JavaScript-heavy sites explains how to tell.

Step 4: get JSON instead of HTML

Parsing HTML yourself is where most scrapers break. Instead, describe the fields you want with extract_rules: a JSON object mapping each field name to a CSS selector and what to return from it.

output	Returns
text	The element’s text
html	The element’s inner HTML
@href, @src and so on	An attribute’s value

For example, {"title": {"selector": "h1", "output": "text"}} returns {"title": "Example Domain"} for example.com. Many pages also embed their data as structured JSON already; our guide to extracting JSON-LD instead of parsing HTML shows how to use it.

Step 5: handle errors properly

Every scraper meets failures: slow sites, pages that will not render, too many requests at once. The API tells you what went wrong with the status code, and the right response depends on which one you get:

Status	Meaning	What to do
200	Success	Use the response
400, 401, 403	A problem with your request, key or plan features	Fix it; retrying will not help
408	The target site timed out	Retry, perhaps with a higher timeout
422	The page failed to render	Retry; check the site works in a normal browser
429	Too many concurrent requests for your plan	Wait, then retry
500	An internal error	Retry after a short delay

On Shifter, failed requests are not charged; you pay one credit for each successful response. The API also retries some failures internally before answering, and those count as one call.

Putting it together

Here is a small Python helper that wraps all five steps: it reads the key from the environment, adds rendering and extraction when you ask for them, retries the errors worth retrying with a growing pause, and stops immediately on the ones that need a fix.

import json
import os
import time

import requests

API = "https://scrape.shifter.io/v1"
RETRY = {408, 422, 429, 500}          # worth another try after a pause
STOP = {400, 401, 403}                # fix the request; retrying will not help


def scrape(url, render_js=False, extract_rules=None, attempts=3, session=None, wait=2.0):
    """Fetch one page through the Web Scraping API; returns text, or parsed JSON with extract_rules."""
    session = session or requests.Session()
    params = {"api_key": os.environ["SHIFTER_SCRAPER_KEY"], "url": url}
    if render_js:
        params["render_js"] = 1
    if extract_rules:
        params["extract_rules"] = json.dumps(extract_rules)
    for attempt in range(1, attempts + 1):
        response = session.get(API, params=params, timeout=120)
        if response.status_code == 200:
            return response.json() if extract_rules else response.text
        if response.status_code in STOP or attempt == attempts or response.status_code not in RETRY:
            raise RuntimeError(f"{url}: HTTP {response.status_code} {response.text[:200]}")
        time.sleep(wait * attempt)    # back off a little more each time

Using it:

from scrape_api import scrape

html = scrape("https://example.com")
data = scrape("https://example.com", extract_rules={"title": {"selector": "h1", "output": "text"}})
print(data["title"])

We tested the helper’s request building, retries and stop conditions against simulated responses for each status code above.

If you are new to retry logic, our guide to retry and backoff for web scrapers explains why the pause grows with each attempt.

Step 6: scale up

Once single requests work, three things matter as you grow:

• Concurrency. Send several requests at once with a thread pool or an async client. Each plan has a concurrency limit (20 to 500 requests at once on Shifter, depending on the plan); stay under it and treat 429 as a signal to slow down.
• Location and proxy type. Use the country parameter to fetch a page as it appears in a specific country. On Growth plans and above, premium_proxy routes requests through residential IPs for sites that block data-centre traffic.
• Sessions. For multi-step flows, a session_id keeps the same IP and cookies across requests.

When the data starts flowing, store it somewhere you can query. Our guide to moving web scraping API data into SQL covers a clean pipeline from response to table.

Common beginner mistakes

• Hard-coding the API key. Use an environment variable or a secrets manager.
• Rendering everything. Try a plain request first; render only pages that need it.
• Retrying everything. A 401 will fail forever. Retry only the errors that can succeed later.
• Parsing HTML by hand. Use extraction rules or embedded JSON where you can.
• Ignoring concurrency limits. Too many parallel requests produce 429s, not speed.

FAQ

What is a web scraping API?

A service that fetches web pages for you through its own proxies and browsers, handles retries and blocks, and returns the HTML or extracted data in response to a single request.

Do I need to know how to code to use a web scraping API?

A little. A single request works from the command line with curl, but most real projects use a short script in Python or another language to send requests, handle errors and store the results.

When should I enable JavaScript rendering?

When the HTML returned without rendering is missing content you can see in your browser. Many sites build their pages with JavaScript, and rendering returns the page as a browser would show it.

Do failed requests cost credits?

On Shifter, no. You are charged one credit per successful response, and failed requests, including errors from the target site, are free.

The bottom line

Using a web scraping API comes down to a few habits: keep the key in the environment, start with plain requests and add rendering only where needed, ask for JSON with extraction rules instead of parsing HTML, retry only the errors that can succeed later, and respect your plan’s concurrency. The helper above covers all of them in about twenty lines. Plans and limits are on the Web Scraping API pricing page.