Start Free TrialLog In
Integration

Use Shifter with Apify

Drop Shifter's residential and ISP proxies into any Apify Actor — Crawlee handles the queueing and retries, Shifter handles the residential IPs. ProxyConfiguration accepts Shifter URLs natively.

Quick Start

Install

npm install apify crawlee

Basic Usage

// main.js (an Apify Actor)
import { Actor } from "apify";
import { CheerioCrawler, ProxyConfiguration } from "crawlee";

await Actor.init();

const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: [
    "customer-USERNAME-country-us-sid-123ABC:PASSWORD@p.shifter.io:443",
  ],
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  async requestHandler({ request, $, log }) {
    log.info(`${request.url} -> ${$("h1").text().trim()}`);
  },
});

await crawler.run(["https://example.com"]);
await Actor.exit();

Features

Native ProxyConfiguration support in Crawlee — pass a Shifter URL or a newUrlFunction
Compatible with CheerioCrawler, PuppeteerCrawler, PlaywrightCrawler, and BasicCrawler
Per-session sticky IPs via Crawlee's session pool — bans automatically retire stale sessions
Geo-targeting in 195+ countries — parameterize country via Actor input schema
Drop-in for Apify Console scheduled runs, Apify Cloud auto-scaling, and self-hosted Crawlee
Compatible with the entire Apify SDK — pushData, key-value store, request queue, and dataset persistence

Examples

Crawlee + Per-Session Rotation

Crawlee handles bans and session expiry automatically. Use newUrlFunction to mint a fresh Shifter URL per session — when Crawlee retires a session due to a ban, the next one gets a clean residential IP.

import { Actor } from "apify";
import { CheerioCrawler, ProxyConfiguration } from "crawlee";

await Actor.init();

const proxyConfiguration = new ProxyConfiguration({
  // Each session asks for a fresh URL — and Crawlee bumps the session
  // on bans, so stale IPs get cycled out automatically.
  newUrlFunction: () => {
    const sid = Math.random().toString(36).slice(2, 10);
    return `customer-USERNAME-country-uk-sid-${sid}-ttl-300:PASSWORD@p.shifter.io:443`;
  },
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  useSessionPool: true,
  persistCookiesPerSession: true,
  maxConcurrency: 8,

  async requestHandler({ request, $, enqueueLinks, log, session }) {
    log.info(`Session ${session.id} -> ${request.url}`);

    $(".product-card").each((_, el) => {
      // Push to dataset (auto-persisted by Apify)
      Actor.pushData({
        url:   request.url,
        title: $(el).find("h2").text().trim(),
        price: $(el).find(".price").text().trim(),
      });
    });

    await enqueueLinks({ selector: "a.next-page", strategy: "same-domain" });
  },

  failedRequestHandler({ request, log }) {
    log.error(`Failed after retries: ${request.url}`);
  },
});

await crawler.run(["https://example.co.uk/products"]);
await Actor.exit();

PuppeteerCrawler (JS-Heavy Targets)

When the target needs a real browser, swap CheerioCrawler for PuppeteerCrawler. The same ProxyConfiguration plugs in — Crawlee passes the Shifter URL into Puppeteer's launch args.

import { Actor } from "apify";
import { PuppeteerCrawler, ProxyConfiguration } from "crawlee";

await Actor.init();

const proxyConfiguration = new ProxyConfiguration({
  newUrlFunction: () => {
    const sid = Math.random().toString(36).slice(2, 10);
    return `customer-USERNAME-country-de-city-berlin-sid-${sid}:PASSWORD@p.shifter.io:443`;
  },
});

const crawler = new PuppeteerCrawler({
  proxyConfiguration,
  useSessionPool: true,
  launchContext: {
    launchOptions: { headless: "new" },
  },
  maxConcurrency: 4,

  async requestHandler({ request, page, log }) {
    log.info(`Visiting ${request.url}`);
    await page.waitForSelector(".product");

    const products = await page.$$eval(".product", (els) =>
      els.map((el) => ({
        title: el.querySelector("h2")?.textContent?.trim(),
        price: el.querySelector(".price")?.textContent?.trim(),
      })),
    );

    await Actor.pushData(products);
  },
});

await crawler.run(["https://example.de/categories/electronics"]);
await Actor.exit();

Per-Country Actor with Input Schema

Expose country as an Apify Actor input. The Actor reads it at startup and configures Shifter for the matching residential pool. Same Actor code works for every region.

// .actor/input_schema.json
{
  "title": "Localized Scraper Input",
  "type": "object",
  "schemaVersion": 1,
  "properties": {
    "startUrl": { "type": "string", "title": "Start URL", "default": "https://example.com" },
    "country":  { "type": "string", "title": "Country",   "enum": ["us","uk","de","jp","fr","br"], "default": "us" },
    "maxPages": { "type": "integer", "title": "Max Pages", "default": 100, "minimum": 1, "maximum": 5000 }
  },
  "required": ["startUrl", "country"]
}

// main.js
import { Actor } from "apify";
import { CheerioCrawler, ProxyConfiguration } from "crawlee";

await Actor.init();

const { startUrl, country, maxPages } = await Actor.getInput();

const proxyConfiguration = new ProxyConfiguration({
  newUrlFunction: () => {
    const sid = Math.random().toString(36).slice(2, 10);
    return `customer-USERNAME-country-${country}-sid-${sid}-ttl-300:PASSWORD@p.shifter.io:443`;
  },
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  maxRequestsPerCrawl: maxPages,
  useSessionPool: true,

  async requestHandler({ request, $, enqueueLinks }) {
    await Actor.pushData({
      country,
      url:   request.url,
      title: $("title").text().trim(),
      h1:    $("h1").first().text().trim(),
    });
    await enqueueLinks({ strategy: "same-domain" });
  },
});

await crawler.run([startUrl]);
await Actor.exit();

Apify SDK Outside Crawlee (Custom Logic)

If Crawlee doesn't fit your shape, you can still pull a Shifter proxy URL from ProxyConfiguration and use it with any HTTP client. Sessions, retries, and persistence all still work.

import { Actor } from "apify";
import { ProxyConfiguration } from "crawlee";
import { gotScraping } from "got-scraping";

await Actor.init();

const proxyConfiguration = new ProxyConfiguration({
  newUrlFunction: () => {
    const sid = Math.random().toString(36).slice(2, 10);
    return `customer-USERNAME-country-fr-sid-${sid}:PASSWORD@p.shifter.io:443`;
  },
});

// Pull a fresh proxy URL per logical task
async function fetchTarget(url) {
  const proxyUrl = await proxyConfiguration.newUrl();

  const html = await gotScraping({
    url,
    proxyUrl,
    headerGeneratorOptions: {
      browsers: [{ name: "chrome", minVersion: 120 }],
      locales:  ["en-US"],
    },
  }).text();

  return html;
}

const urls = [
  "https://example.fr/api/v1/products?page=1",
  "https://example.fr/api/v1/products?page=2",
  // ...
];

for (const url of urls) {
  try {
    const html = await fetchTarget(url);
    await Actor.pushData({ url, length: html.length });
  } catch (err) {
    console.error(`Failed ${url}: ${err.message}`);
  }
}

await Actor.exit();
FAQ

Frequently asked questions

Common questions about using Shifter with Apify.

Use Crawlee's ProxyConfiguration class with either a `proxyUrls` array (one or more Shifter URLs) or a `newUrlFunction` that returns a fresh Shifter URL per session. Pass the configuration to your crawler — every request routes through Shifter automatically.

Apify Proxy and Shifter are independent products. To use Shifter, skip the `useApifyProxy` option and provide your own ProxyConfiguration with Shifter URLs. You can still mix them — some Actors use Apify Proxy for warm-up and Shifter for production scrapes.

Enable `useSessionPool: true` on your crawler. Crawlee will tag each request with a session and reuse the same proxy URL (and therefore the same Shifter sid) across requests in that session. When a session triggers a ban, Crawlee retires it and starts a fresh one with a new IP.

Yes. Define a `country` field in your Actor's input_schema.json and read it at startup with `Actor.getInput()`. Bake it into the Shifter username (e.g. `country-${input.country}`) inside your newUrlFunction. The same Actor code works for every region.

Yes. ProxyConfiguration plugs into all four Crawlee crawlers — Cheerio, JsdomCrawler, Puppeteer, and Playwright. Crawlee handles the proxy negotiation transparently, including authenticating in headless Chromium.

Yes. Apify Console runs your Actor source in their cloud. Store Shifter credentials as Actor secrets (Settings > Environment variables), reference them in your code via `process.env.SHIFTER_USER` etc., and the Actor routes through Shifter on every scheduled run.
Get started

Start Using Shifter with Apify

Run Apify Actors through Shifter's 205M+ residential and ISP proxies. Native Crawlee ProxyConfiguration, per-session sticky IPs, and full Cheerio / Puppeteer / Playwright crawler support.

Try Shifter for FreeSet up in minutes. Cancel anytime.