Start Free TrialLog In
Integration

Use Shifter with Apify

Drop Shifter's residential and ISP proxies into any Apify Actor — Crawlee handles the queueing and retries, Shifter handles the residential IPs. ProxyConfiguration accepts Shifter URLs natively.

Quick Start

Install

npm install apify crawlee

Basic Usage

// main.js (an Apify Actor)
import { Actor } from "apify";
import { CheerioCrawler, ProxyConfiguration } from "crawlee";

await Actor.init();

const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: [
    "customer-USERNAME-country-us-sid-123ABC:PASSWORD@p.shifter.io:443",
  ],
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  async requestHandler({ request, $, log }) {
    log.info(`${request.url} -> ${$("h1").text().trim()}`);
  },
});

await crawler.run(["https://example.com"]);
await Actor.exit();

Features

Native ProxyConfiguration support in Crawlee — pass a Shifter URL or a newUrlFunction
Compatible with CheerioCrawler, PuppeteerCrawler, PlaywrightCrawler, and BasicCrawler
Per-session sticky IPs via Crawlee's session pool — bans automatically retire stale sessions
Geo-targeting in 195+ countries — parameterize country via Actor input schema
Drop-in for Apify Console scheduled runs, Apify Cloud auto-scaling, and self-hosted Crawlee
Compatible with the entire Apify SDK — pushData, key-value store, request queue, and dataset persistence

Examples

Crawlee + Per-Session Rotation

Crawlee handles bans and session expiry automatically. Use newUrlFunction to mint a fresh Shifter URL per session — when Crawlee retires a session due to a ban, the next one gets a clean residential IP.

import { Actor } from "apify";
import { CheerioCrawler, ProxyConfiguration } from "crawlee";

await Actor.init();

const proxyConfiguration = new ProxyConfiguration({
  // Each session asks for a fresh URL — and Crawlee bumps the session
  // on bans, so stale IPs get cycled out automatically.
  newUrlFunction: () => {
    const sid = Math.random().toString(36).slice(2, 10);
    return `customer-USERNAME-country-uk-sid-${sid}-ttl-300:PASSWORD@p.shifter.io:443`;
  },
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  useSessionPool: true,
  persistCookiesPerSession: true,
  maxConcurrency: 8,

  async requestHandler({ request, $, enqueueLinks, log, session }) {
    log.info(`Session ${session.id} -> ${request.url}`);

    $(".product-card").each((_, el) => {
      // Push to dataset (auto-persisted by Apify)
      Actor.pushData({
        url:   request.url,
        title: $(el).find("h2").text().trim(),
        price: $(el).find(".price").text().trim(),
      });
    });

    await enqueueLinks({ selector: "a.next-page", strategy: "same-domain" });
  },

  failedRequestHandler({ request, log }) {
    log.error(`Failed after retries: ${request.url}`);
  },
});

await crawler.run(["https://example.co.uk/products"]);
await Actor.exit();

PuppeteerCrawler (JS-Heavy Targets)

When the target needs a real browser, swap CheerioCrawler for PuppeteerCrawler. The same ProxyConfiguration plugs in — Crawlee passes the Shifter URL into Puppeteer's launch args.

import { Actor } from "apify";
import { PuppeteerCrawler, ProxyConfiguration } from "crawlee";

await Actor.init();

const proxyConfiguration = new ProxyConfiguration({
  newUrlFunction: () => {
    const sid = Math.random().toString(36).slice(2, 10);
    return `customer-USERNAME-country-de-city-berlin-sid-${sid}:PASSWORD@p.shifter.io:443`;
  },
});

const crawler = new PuppeteerCrawler({
  proxyConfiguration,
  useSessionPool: true,
  launchContext: {
    launchOptions: { headless: "new" },
  },
  maxConcurrency: 4,

  async requestHandler({ request, page, log }) {
    log.info(`Visiting ${request.url}`);
    await page.waitForSelector(".product");

    const products = await page.$$eval(".product", (els) =>
      els.map((el) => ({
        title: el.querySelector("h2")?.textContent?.trim(),
        price: el.querySelector(".price")?.textContent?.trim(),
      })),
    );

    await Actor.pushData(products);
  },
});

await crawler.run(["https://example.de/categories/electronics"]);
await Actor.exit();

Per-Country Actor with Input Schema

Expose country as an Apify Actor input. The Actor reads it at startup and configures Shifter for the matching residential pool. Same Actor code works for every region.

// .actor/input_schema.json
{
  "title": "Localized Scraper Input",
  "type": "object",
  "schemaVersion": 1,
  "properties": {
    "startUrl": { "type": "string", "title": "Start URL", "default": "https://example.com" },
    "country":  { "type": "string", "title": "Country",   "enum": ["us","uk","de","jp","fr","br"], "default": "us" },
    "maxPages": { "type": "integer", "title": "Max Pages", "default": 100, "minimum": 1, "maximum": 5000 }
  },
  "required": ["startUrl", "country"]
}

// main.js
import { Actor } from "apify";
import { CheerioCrawler, ProxyConfiguration } from "crawlee";

await Actor.init();

const { startUrl, country, maxPages } = await Actor.getInput();

const proxyConfiguration = new ProxyConfiguration({
  newUrlFunction: () => {
    const sid = Math.random().toString(36).slice(2, 10);
    return `customer-USERNAME-country-${country}-sid-${sid}-ttl-300:PASSWORD@p.shifter.io:443`;
  },
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  maxRequestsPerCrawl: maxPages,
  useSessionPool: true,

  async requestHandler({ request, $, enqueueLinks }) {
    await Actor.pushData({
      country,
      url:   request.url,
      title: $("title").text().trim(),
      h1:    $("h1").first().text().trim(),
    });
    await enqueueLinks({ strategy: "same-domain" });
  },
});

await crawler.run([startUrl]);
await Actor.exit();

Apify SDK Outside Crawlee (Custom Logic)

If Crawlee doesn't fit your shape, you can still pull a Shifter proxy URL from ProxyConfiguration and use it with any HTTP client. Sessions, retries, and persistence all still work.

import { Actor } from "apify";
import { ProxyConfiguration } from "crawlee";
import { gotScraping } from "got-scraping";

await Actor.init();

const proxyConfiguration = new ProxyConfiguration({
  newUrlFunction: () => {
    const sid = Math.random().toString(36).slice(2, 10);
    return `customer-USERNAME-country-fr-sid-${sid}:PASSWORD@p.shifter.io:443`;
  },
});

// Pull a fresh proxy URL per logical task
async function fetchTarget(url) {
  const proxyUrl = await proxyConfiguration.newUrl();

  const html = await gotScraping({
    url,
    proxyUrl,
    headerGeneratorOptions: {
      browsers: [{ name: "chrome", minVersion: 120 }],
      locales:  ["en-US"],
    },
  }).text();

  return html;
}

const urls = [
  "https://example.fr/api/v1/products?page=1",
  "https://example.fr/api/v1/products?page=2",
  // ...
];

for (const url of urls) {
  try {
    const html = await fetchTarget(url);
    await Actor.pushData({ url, length: html.length });
  } catch (err) {
    console.error(`Failed ${url}: ${err.message}`);
  }
}

await Actor.exit();
FAQ

Frequently asked FAQ questions

Common questions about using Shifter with Apify.

Use Crawlee's ProxyConfiguration class with either a `proxyUrls` array (one or more Shifter URLs) or a `newUrlFunction` that returns a fresh Shifter URL per session. Pass the configuration to your crawler — every request routes through Shifter automatically.
Get started

Start Using Shifter with Apify

Run Apify Actors through Shifter's 205M+ residential and ISP proxies. Native Crawlee ProxyConfiguration, per-session sticky IPs, and full Cheerio / Puppeteer / Playwright crawler support.

Try Shifter for FreeSet up in minutes. Cancel anytime.