Web Scraping API
One endpoint, every page. The scraping API handles proxy rotation, headless Chrome, JavaScript execution, CAPTCHA solving, and automatic retries.
Endpoint
Section titled “Endpoint” GET https://scrape.shifter.io/v1
POST and PUT variants are also supported, see POST requests and PUT requests.
Authentication
Section titled “Authentication”Pass your API key as the api_key query parameter on every request. Keys are managed in the panel under Web Scraping API → API Keys.
curl "https://scrape.shifter.io/v1?api_key=YOUR_API_KEY&url=https://example.com"Core parameters
Section titled “Core parameters”| Parameter | Type | Required | Description |
|---|---|---|---|
api_key | string | yes | Your API key |
url | string | yes | The URL to fetch |
render_js | 0 | 1 | no | Enable headless Chrome rendering |
premium_proxy | 0 | 1 | no | Route through residential IPs (Growth+) |
mobile_proxy | 0 | 1 | no | Route through mobile IPs (Growth+) |
country | ISO code | no | Country for proxy geolocation |
device | desktop | mobile | tablet | no | Simulated device |
session_id | string | no | Reuse cookies and proxy across requests |
timeout | ms | no | Max time the browser can spend, up to 60000 |
wait_for | ms | no | Idle wait after navigation (5000-10000 recommended) |
wait_for_css | selector | no | Block until CSS selector appears |
wait_until | string | no | Navigation wait condition (advanced) |
keep_headers | 0 | 1 | no | Forward custom headers to target |
custom_headers | JSON | no | Custom request headers (URL-encoded) |
custom_cookies | string | no | Cookie header value |
extract_rules | JSON | no | CSS selectors to parse into structured JSON |
auto_parser | 0 | 1 | no | Auto-parse JSON responses |
screenshot | 0 | 1 | no | Return PNG instead of HTML |
screenshot_options | JSON | no | Screenshot size and selector |
binary_response | 0 | 1 | no | Stream binary payloads (PDFs, images) |
js_instructions | JSON | no | Run scroll/click/wait before capture |
disable_stealth | 0 | 1 | no | Disable stealth plugin |
webhook | URL | no | POST the response to this URL when ready |
forward_proxy | URL | no | Chain through your upstream proxy |
Every parameter that expects JSON must be URL-encoded.
Quick example
Section titled “Quick example”curl "https://scrape.shifter.io/v1?api_key=YOUR_API_KEY&url=https://example.com&render_js=1&wait_for_css=h1"Endpoint pages
Section titled “Endpoint pages” Basic request Minimal GET with api_key and url.
Full parameters Every parameter documented in detail.
POST requests Scrape URLs that require POST bodies.
PUT requests Scrape PUT endpoints.
Rendering JavaScript render_js flag and limits.
JavaScript instructions Scroll, click, wait before capture.
Waiting for CSS wait_for_css selector syntax.
Forcing timeouts Cap browser time with timeout.
Screenshot Capture full-page or element PNG.
Sessions Hold cookies and IP across requests.
Binary files Download PDFs, images, other binaries.
Custom headers Forward headers to the target.
Custom cookies Inject cookies into the request.
Proxies Datacenter vs residential vs mobile.
Geolocation country parameter and supported locations.
Extraction rules Return parsed JSON.
JSON auto-parser auto_parser for JSON endpoints.
API errors Status codes and retry semantics.