Start Free TrialLog In
Integration

Use Shifter with Cheerio

Pair Shifter's residential and ISP proxies with Cheerio for fast, lean Node scraping. Cheerio handles the jQuery-style HTML parsing, Shifter handles the residential IPs — no headless browser required.

Quick Start

Install

npm install cheerio axios https-proxy-agent

Basic Usage

import axios from "axios";
import { HttpsProxyAgent } from "https-proxy-agent";
import { load } from "cheerio";

const proxyUrl =
  "customer-USERNAME-country-us-sid-123ABC:PASSWORD@p.shifter.io:443";

const { data: html } = await axios.get("https://example.com", {
  httpsAgent: new HttpsProxyAgent(proxyUrl),
  proxy: false,
});

const $ = load(html);
console.log($("h1").text());

$("article.post").each((_, el) => {
  console.log($(el).find("h2").text(), "->", $(el).find("a").attr("href"));
});

Features

Pairs cleanly with axios, got, got-scraping, undici, and any HTTP client that supports a proxy URL
Per-request rotation by default, with `sid` for sticky sessions and `ttl-N` for timed pins of N seconds
Compatible with Crawlee, Apify, and any production crawler framework that accepts a proxy URL
Geo-targeting in 195+ countries via username parameters — country, region, city, ASN
Order-of-magnitude faster than headless-browser scraping for static or JS-light targets
Compatible with TypeScript, ESM, CommonJS, and every Node.js LTS from 14 onwards

Examples

Sticky Session + Multi-Page Crawl

Pin one residential IP for the duration of a multi-page crawl. Add `country-uk` to geo-target and `ttl-300` to extend the sticky window to 5 minutes.

import axios, { type AxiosInstance } from "axios";
import { HttpsProxyAgent } from "https-proxy-agent";
import { load } from "cheerio";
import { randomBytes } from "node:crypto";

function makeClient(country: string): AxiosInstance {
  const sid = randomBytes(4).toString("hex");
  const proxyUrl =
    `customer-USERNAME-country-${country}-sid-${sid}-ttl-300:` +
    `PASSWORD@p.shifter.io:443`;

  return axios.create({
    httpsAgent: new HttpsProxyAgent(proxyUrl),
    proxy: false,
    timeout: 30_000,
    headers: {
      "User-Agent":
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
      "Accept-Language": "en-US,en;q=0.9",
    },
  });
}

const client = makeClient("uk");

let url: string | null = "https://example.co.uk/products";
const products: { title: string; price: string }[] = [];

while (url) {
  const { data: html } = await client.get(url);
  const $ = load(html);

  $(".product-card").each((_, el) => {
    products.push({
      title: $(el).find("h2").text().trim(),
      price: $(el).find(".price").text().trim(),
    });
  });

  const next = $("a.next-page").attr("href");
  url = next ? new URL(next, url).toString() : null;
}

console.log(`Scraped ${products.length} products`);

Parallel Scraping (per-request rotation)

Drop the sid for per-request rotation. Each parallel fetch gets a different residential IP — perfect for hitting a list of URLs without tripping per-IP rate limits.

import axios from "axios";
import { HttpsProxyAgent } from "https-proxy-agent";
import { load } from "cheerio";

function rotatingClient() {
  // No sid -> Shifter rotates the residential IP per request.
  const proxyUrl =
    "customer-USERNAME-country-us:PASSWORD@p.shifter.io:443";

  return axios.create({
    httpsAgent: new HttpsProxyAgent(proxyUrl),
    proxy: false,
    timeout: 30_000,
  });
}

async function scrape(url: string) {
  const client = rotatingClient();
  const { data: html } = await client.get(url);
  const $ = load(html);

  return {
    url,
    title: $("h1").first().text().trim(),
    headings: $("h2").map((_, el) => $(el).text().trim()).get(),
  };
}

const urls = [
  "https://example.com/category/laptops",
  "https://example.com/category/phones",
  "https://example.com/category/tablets",
  "https://example.com/category/wearables",
];

const results = await Promise.all(urls.map(scrape));
console.log(results);

got-scraping (built-in browser fingerprints)

got-scraping bundles realistic header generation on top of got. Pair with Shifter for a tiny, browser-shaped HTTP scraper that's still 100x faster than Puppeteer.

// npm install got-scraping cheerio
import { gotScraping } from "got-scraping";
import { load } from "cheerio";

const proxyUrl =
  "customer-USERNAME-country-de-city-berlin-sid-456DEF:PASSWORD@p.shifter.io:443";

const html = await gotScraping({
  url: "https://example.de/products",
  proxyUrl,
  headerGeneratorOptions: {
    browsers: [{ name: "chrome", minVersion: 120 }],
    devices: ["desktop"],
    locales: ["de-DE", "en"],
    operatingSystems: ["macos", "linux"],
  },
}).text();

const $ = load(html);

$(".product").each((_, el) => {
  console.log({
    title: $(el).find("h2").text().trim(),
    price: $(el).find(".price").text().trim(),
  });
});

Crawlee (production crawler with Shifter)

Crawlee (by Apify) handles queues, retries, persistence, and proxy rotation out of the box. Plug Shifter as a ProxyConfiguration and let Crawlee orchestrate the rest.

// npm install crawlee cheerio
import { CheerioCrawler, ProxyConfiguration } from "crawlee";

const proxyConfiguration = new ProxyConfiguration({
  newUrlFunction: () => {
    // New residential IP per session. Crawlee rotates sessions
    // automatically on bans, so stale IPs get cycled out.
    const sid = Math.random().toString(36).slice(2, 10);
    return `customer-USERNAME-country-fr-sid-${sid}:PASSWORD@p.shifter.io:443`;
  },
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  maxConcurrency: 8,
  async requestHandler({ request, $, enqueueLinks, log }) {
    log.info(`${request.url} -> ${$("h1").text().trim()}`);

    $(".article").each((_, el) => {
      log.info(`  ${$(el).find("h2").text().trim()}`);
    });

    await enqueueLinks({ selector: "a.next-page", strategy: "same-domain" });
  },
});

await crawler.run([
  "https://example.fr/blog",
  "https://example.fr/news",
]);
FAQ

Frequently asked questions

Common questions about using Shifter with Cheerio.

No. Cheerio is a parser — it doesn't make HTTP requests. The proxy is configured on whichever HTTP client you pair with Cheerio (axios, got, undici, fetch). Once the HTML is fetched through Shifter, you pass it to Cheerio's load() function as usual.

Configure axios with an https-proxy-agent pointing at p.shifter.io:443 (with credentials inline). Fetch the page, pass response.data to cheerio.load(), then query as you would with jQuery. The full setup is about 10 lines of code.

Use Cheerio whenever the data you need is in the initial HTML payload — it's 10-100x faster than headless Chrome and uses a tiny fraction of the memory. Switch to Puppeteer or Playwright only when the page hydrates content client-side via JavaScript.

Add a session ID to the proxy username — for example `customer-USERNAME-country-us-sid-123ABC`. Reuse the same axios instance across every fetch in the crawl run and Shifter will keep returning the same residential IP. Add `ttl-N` to extend the lifetime.

Yes. Crawlee ships with a CheerioCrawler that handles queueing, retries, and concurrency. Configure Shifter via Crawlee's ProxyConfiguration with a newUrlFunction that returns a fresh Shifter URL with a unique sid per session.

Yes. got-scraping accepts a `proxyUrl` option and handles the proxy negotiation transparently. Combined with its built-in header generator (which mimics real browser fingerprints), it's an excellent lightweight alternative to headless browsers.
Get started

Start Using Shifter with Cheerio

Pair Shifter's 205M+ residential and ISP proxies with Cheerio for fast, lean Node scraping. Per-request rotation, sticky sessions, and full Crawlee support.

Try Shifter for FreeSet up in minutes. Cancel anytime.