Enhancing PHP Applications with Web Scraping API Integration

A lot of the web still runs on PHP: shops, CMSs, internal tools and customer portals. When one of those applications needs data from other sites, such as competitor prices, product details or listings, the tempting route is to fetch pages with file_get_contents and parse them with regular expressions. It works on day one and breaks on day ten, when the site blocks the server’s IP or moves its markup.

A web scraping API removes most of that fragility. It fetches pages through its own proxies and browsers, retries failures and can return just the fields you ask for as JSON. This tutorial builds a small PHP client for one, with no dependencies beyond the curl extension, and shows how to use it in an application.

What the client needs to do

Four things cover most real use:

1. Send a request with the API key and the target URL.
2. Ask for structured data with extraction rules, so the application never parses HTML.
3. Retry the right errors and stop on the ones that will never succeed.
4. Fetch many pages in parallel without overrunning the plan’s concurrency limit.

If you have not used a scraping API before, our beginner’s walkthrough covers the request format and status codes in more detail.

The client

<?php

final class ShifterScraper
{
    private const RETRY = [408, 422, 429, 500];

    public function __construct(
        private string $apiKey,
        private string $endpoint = 'https://scrape.shifter.io/v1',
        private int $attempts = 3,
        private float $backoffSeconds = 2.0,
    ) {}

    /** Fetch one page; with $extractRules, returns the decoded JSON instead of HTML. */
    public function fetch(string $url, array $extractRules = [], bool $renderJs = false): string|array
    {
        $params = ['api_key' => $this->apiKey, 'url' => $url];
        if ($renderJs) {
            $params['render_js'] = 1;
        }
        if ($extractRules) {
            $params['extract_rules'] = json_encode($extractRules);
        }

        for ($attempt = 1; ; $attempt++) {
            [$status, $body] = $this->get($this->endpoint . '?' . http_build_query($params));
            if ($status === 200) {
                return $extractRules ? json_decode($body, true, flags: JSON_THROW_ON_ERROR) : $body;
            }
            if (!in_array($status, self::RETRY, true) || $attempt >= $this->attempts) {
                throw new RuntimeException("HTTP {$status} for {$url}: " . substr($body, 0, 200));
            }
            usleep((int) ($this->backoffSeconds * $attempt * 1_000_000));
        }
    }

    /** Fetch many pages at once with curl_multi, at most $concurrency in flight. */
    public function fetchMany(array $urls, array $extractRules = [], int $concurrency = 10): array
    {
        $results = [];
        foreach (array_chunk($urls, $concurrency) as $batch) {
            $multi = curl_multi_init();
            $handles = [];
            foreach ($batch as $url) {
                $params = ['api_key' => $this->apiKey, 'url' => $url];
                if ($extractRules) {
                    $params['extract_rules'] = json_encode($extractRules);
                }
                $handles[$url] = $this->handle($this->endpoint . '?' . http_build_query($params));
                curl_multi_add_handle($multi, $handles[$url]);
            }
            do {
                curl_multi_exec($multi, $running);
                curl_multi_select($multi);
            } while ($running > 0);
            foreach ($handles as $url => $h) {
                $status = curl_getinfo($h, CURLINFO_RESPONSE_CODE);
                $body = curl_multi_getcontent($h);
                $results[$url] = $status === 200
                    ? ($extractRules ? json_decode($body, true) : $body)
                    : $this->fetchOrNull($url, $extractRules);   // retry failures one by one
                curl_multi_remove_handle($multi, $h);
            }
            curl_multi_close($multi);
        }
        return $results;
    }

    private function fetchOrNull(string $url, array $extractRules): string|array|null
    {
        try {
            return $this->fetch($url, $extractRules);
        } catch (RuntimeException) {
            return null;
        }
    }

    private function handle(string $requestUrl): CurlHandle
    {
        $h = curl_init($requestUrl);
        curl_setopt_array($h, [CURLOPT_RETURNTRANSFER => true, CURLOPT_TIMEOUT => 120]);
        return $h;
    }

    private function get(string $requestUrl): array
    {
        $h = $this->handle($requestUrl);
        $body = curl_exec($h);
        $status = curl_getinfo($h, CURLINFO_RESPONSE_CODE);
        return [$status, $body === false ? curl_error($h) : $body];
    }
}

A few design choices worth calling out:

• Retries only where they help. Timeouts (408), failed renders (422), rate limits (429) and server errors (500) are retried with a growing pause. Anything else, such as an invalid key, throws immediately.
• Extraction rules become an array. With rules, fetch() returns decoded JSON, so calling code works with $product['price'], not with markup.
• Parallel requests with curl_multi. fetchMany() sends a batch of requests at once and waits for all of them. Batches are sized to stay under your plan’s concurrency limit. Pages that fail in the batch get a second chance through fetch() with its retries, and come back as null if they still fail.

Using it

<?php
require __DIR__ . '/ShifterScraper.php';

$scraper = new ShifterScraper(getenv('SHIFTER_SCRAPER_KEY'));

// One page, as structured data instead of HTML
$product = $scraper->fetch('https://shop.example.com/p/123', [
    'name'  => ['selector' => 'h1', 'output' => 'text'],
    'price' => ['selector' => '.price', 'output' => 'text'],
]);

// A batch, ten requests in flight at a time; failed pages come back as null
$pages = $scraper->fetchMany($productUrls, ['name' => ['selector' => 'h1', 'output' => 'text']], 10);

Keep the API key in an environment variable or your framework’s secrets configuration, never in the code.

How we tested it

We ran the client against a local PHP server standing in for the API. It served normal pages, one page that failed twice with 429 before succeeding, one that always returned 401 and one that always timed out with 408. The tests checked that:

• HTML and extracted JSON came back correctly;
• the flaky page succeeded on its third attempt;
• the 401 was tried once and the 408 was retried up to the cap before failing;
• a batch of 13 pages returned 12 results and one null for the failing page.

All 11 checks passed on PHP 8.5.

Fitting it into an application

Do not scrape during a page view. A user should never wait for a remote site to load. Run collection in a queue worker or a scheduled job, and serve the stored results.

Store results with a timestamp. Save each record with the time it was collected, so the application can show how fresh it is and refresh only what is stale.

Cache aggressively. Most data does not change by the minute. Refreshing hourly or daily is enough for prices and listings on most sites, and every request you skip saves money.

Log failures. Record which URLs come back null and why. A sudden rise usually means a site changed its layout and your extraction rules need updating.

Respect the plan’s limits. Set the batch size to your plan’s concurrency limit or below. On Shifter that limit is between 20 and 500 requests at once, depending on the plan.

When to use proxies directly instead

A scraping API suits most data collection: it handles rendering, retries and IP choice for you. Some jobs still fit a direct proxy connection better, such as when you need full control of the HTTP client, or long sessions on one IP. Our guide to using residential proxies in PHP with cURL and Guzzle covers that route.

FAQ

Can PHP be used for web scraping?

Yes. PHP’s curl extension handles HTTP well, including parallel requests with curl_multi. Pairing it with a web scraping API means the application does not have to manage proxies, browsers or blocks itself.

Do I need Guzzle to call a web scraping API from PHP?

No. The client in this guide uses only the built-in curl extension. Guzzle works too if your application already uses it.

How do I scrape JavaScript-rendered pages from PHP?

Ask the API to render the page by passing render_js, which the client exposes as a parameter. The API runs a browser and returns the rendered page, so PHP never needs a headless browser.

How many requests can I send at once?

As many as your scraping API plan allows. Size fetchMany() batches to that limit or below.

The bottom line

A PHP application can use web data without building scraping infrastructure: a small curl-based client, extraction rules instead of HTML parsing, retries only for the errors that can pass, and parallel batches sized to your plan. Run it from a queue, store results with timestamps, and the application gets fresh data without ever making a user wait for someone else’s website.