Building Scalable Data Pipelines With Node.js and Web Scraping APIs

Fetching one page with a web scraping API is a single line of code. Fetching fifty thousand pages every night, without overrunning your plan’s limits, losing work when the process crashes or silently dropping failures, is an engineering problem. Node.js is a good fit for it: the work is almost all waiting on the network, which is exactly what Node’s event loop does well.

This guide builds a small, production-shaped pipeline in Node.js on top of Shifter’s Web Scraping API: bounded concurrency, retries for the right errors, resumable output and a separate error log. The whole thing is about sixty lines with no dependencies, and we tested it against a mock of the API. If you have not made a first request yet, start with our beginner’s walkthrough of a web scraping API.

The shape of the pipeline

A scraping pipeline has four jobs:

1. A queue of URLs to fetch.
2. Workers that take URLs from the queue and call the API, a fixed number at a time.
3. Error handling that retries what can succeed later and records what cannot.
4. An output that survives crashes and lets a rerun skip finished work.

The API takes care of proxies, browsers and some retries on its side. The pipeline’s job is to feed it at the right rate and keep a reliable record of what happened.

Concurrency: workers, not Promise.all over everything

The most common Node.js mistake is to map every URL to a request and await Promise.all(...) on the lot. With fifty thousand URLs, that fires fifty thousand requests at once. The API answers most of them with 429 Too Many Requests, because every plan has a concurrency limit (on Shifter, from 20 to 500 requests at once depending on the plan).

The fix is a fixed pool of workers sharing one queue. Each worker takes the next URL, processes it, and comes back for another, so exactly concurrency requests are in flight at any moment. No library is needed; a shared array and a loop do it.

Retries: only for errors that can succeed later

Not every error deserves a retry. A timeout, a failed render, a 429 or a server error can succeed on the next attempt. A bad API key or a malformed request will fail forever, and retrying it just wastes time. The pipeline retries 408, 422, 429 and 500 with a pause that grows each attempt, and records everything else as a failure straight away. Our guide to retry and backoff for web scrapers explains why the pause should grow.

Output: append-only and resumable

Each successful result is appended to a newline-delimited JSON (NDJSON) file as soon as it arrives: one line per URL with the data and a timestamp. That has three advantages:

• A crash loses at most the requests in flight. Everything already written stays.
• Reruns are cheap. On start, the pipeline reads the output file and skips any URL already there, so you can restart a failed run without paying for the same pages twice.
• Failures are kept separately in a .errors file, which you can review or feed back into the next run.

Designing so that a rerun produces the same result is the core idea behind idempotent scrapers, which covers it in more depth.

The code

import { appendFile, readFile } from 'node:fs/promises';

const API = process.env.SCRAPER_API_URL ?? 'https://scrape.shifter.io/v1';
const RETRY = new Set([408, 422, 429, 500]);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

// Fetch one URL through the Web Scraping API, retrying only errors that can succeed later.
export async function fetchPage(url, { extractRules, attempts = 3, backoffMs = 2000 } = {}) {
  const params = new URLSearchParams({ api_key: process.env.SHIFTER_SCRAPER_KEY, url });
  if (extractRules) params.set('extract_rules', JSON.stringify(extractRules));
  for (let attempt = 1; ; attempt++) {
    const res = await fetch(`${API}?${params}`, { signal: AbortSignal.timeout(120_000) });
    if (res.ok) return extractRules ? res.json() : res.text();
    if (!RETRY.has(res.status) || attempt === attempts) {
      throw new Error(`HTTP ${res.status} for ${url}`);
    }
    await sleep(backoffMs * attempt);
  }
}

// URLs already written to the output file, so a restarted run skips them.
async function alreadyDone(outFile) {
  try {
    const lines = (await readFile(outFile, 'utf8')).split('\n').filter(Boolean);
    return new Set(lines.map((line) => JSON.parse(line).url));
  } catch {
    return new Set();
  }
}

// Run `concurrency` workers over the queue; each result becomes one NDJSON line.
export async function runPipeline(urls, { outFile, concurrency = 10, extractRules, backoffMs } = {}) {
  const done = await alreadyDone(outFile);
  const queue = urls.filter((u) => !done.has(u));
  const stats = { skipped: urls.length - queue.length, ok: 0, failed: 0 };

  async function worker() {
    for (let url = queue.shift(); url !== undefined; url = queue.shift()) {
      try {
        const data = await fetchPage(url, { extractRules, backoffMs });
        await appendFile(outFile, JSON.stringify({ url, data, at: new Date().toISOString() }) + '\n');
        stats.ok++;
      } catch (err) {
        await appendFile(`${outFile}.errors`, JSON.stringify({ url, error: err.message }) + '\n');
        stats.failed++;
      }
    }
  }

  await Promise.all(Array.from({ length: concurrency }, worker));
  return stats;
}

It uses only Node’s built-in fetch, AbortSignal.timeout and fs/promises, so it runs on any current Node.js release without installing anything.

Running it on a list of product pages, asking the API for JSON rather than HTML:

import { readFile } from 'node:fs/promises';
import { runPipeline } from './pipeline.mjs';

const urls = (await readFile('urls.txt', 'utf8')).split('\n').filter(Boolean);
const stats = await runPipeline(urls, {
  outFile: 'products.ndjson',
  concurrency: 20,                       // stay under your plan's concurrency limit
  extractRules: {
    name: { selector: 'h1', output: 'text' },
    price: { selector: '.price', output: 'text' },
  },
});
console.log(stats);                      // { skipped, ok, failed }

How we tested it

We ran the pipeline against a local stand-in for the API that served 22 URLs: twenty normal pages, one that failed twice with 429 before succeeding, and one that always returned 401. The test checked that:

• 21 pages succeeded and one failure was recorded in the error file;
• the flaky page was retried and succeeded on its third attempt, while the 401 was tried only once;
• no more than five requests were ever in flight with concurrency: 5;
• a second run skipped all 21 finished pages and only retried the failure.

Scaling further

The single-process design above comfortably handles tens of thousands of pages a run. Beyond that:

• Move the queue out of memory. A database table or a message queue lets several processes share the work and survive restarts of the whole machine.
• Load results into a database as they arrive. NDJSON is a good landing format; our guide to moving web scraping API data into SQL covers the loading step.
• Apply backpressure. If the step after fetching is slower than fetching, let it slow the workers down rather than letting results pile up in memory. See backpressure and flow control in distributed crawlers.
• Watch the numbers. Track successes, failures and retries per run, and alert when the failure rate moves. Our guide to monitoring a scraping pipeline lists the metrics that matter.

FAQ

Is Node.js good for web scraping pipelines?

Yes, for pipelines built on an API. The work is mostly waiting for network responses, and Node.js handles many concurrent requests in one process with little overhead. CPU-heavy parsing is better done by the API’s extraction rules or in a separate step.

How many concurrent requests should a Node.js scraper make?

As many as your scraping API plan allows, and no more. Use a fixed pool of workers set to that limit, and treat 429 responses as a signal to back off.

How do I make a scraping job resumable?

Write each result as soon as it arrives to an append-only file or table keyed by URL, and on start skip any URL that already has a result. A crash then costs only the requests that were in flight.

Do I need a queue library?

Not for a single process. A shared array with a fixed number of async workers is enough. Move to a database or message queue when several processes or machines need to share the work.

The bottom line

A scalable scraping pipeline in Node.js needs four things: a fixed pool of workers sized to your plan’s concurrency, retries only for errors that can succeed later, append-only output that a rerun can resume from, and a separate record of failures. The code above does all four in about sixty lines. The API handles proxies and browsers; your pipeline’s job is to feed it steadily and never lose track of what happened.