API request

To scrape a web page, send a GET request to the Web Scraping API base endpoint with your api_key and the target url. Additional optional parameters let you control rendering, proxy type, geolocation, timeouts, and more.

Endpoint

GET https://scrape.shifter.io/v1?api_key=YOUR_API_KEY&url=<TARGET_URL>

Parameters

Parameter	Type	Required	Description
api_key	string	yes	Your Web Scraping API access key. Found in your account dashboard.
url	string	yes	The target URL to scrape.
render_js	integer	no	0 (default) or 1. Enable JavaScript rendering via a headless browser. Costs 5 API calls for datacenter and 25 for residential.
proxy_type	string	no	datacenter (default) or residential. Residential costs 10 calls without JS rendering and 25 with.
country	string	no	2-letter country code for proxy geolocation. Supported list depends on proxy type.
keep_headers	integer	no	1 to pass custom request headers through, 0 to use only the ones you specify.
session	integer	no	Reuse the same proxy IP across multiple requests by reusing the same session number. Expires 5 minutes after last use.
timeout	integer	no	Maximum request time in milliseconds. Max 60000. The API returns whatever HTML was collected before the timeout fires.
device	string	no	desktop (default), mobile, or tablet.
wait_until	string	no	Advanced. Condition for when rendering is considered complete. Requires render_js=1.
wait_for	integer	no	Advanced. Milliseconds to wait after page load. Recommended 5000-10000. Requires render_js=1.
wait_for_css	string	no	Advanced. Wait up to 10 seconds for a CSS selector to appear. Requires render_js=1.
screenshot	integer	no	Advanced. Set to 1 to return a screenshot. Requires render_js=1.
screenshot_options	object	no	Advanced. Controls screenshot dimensions and full_page.
extract_rules	string	no	Advanced. URL-encoded JSON describing CSS extraction rules.
disable_stealth	integer	no	Advanced. Set to 1 to disable the default stealth plugin. Requires render_js=1.
auto_parser	integer	no	Advanced. Set to 1 to return parsed JSON instead of HTML.
js_instructions	string	no	Advanced. URL-encoded JSON array of actions to run before scraping. Requires render_js=1.
binary_response	integer	no	Set to 1 to return a base64-encoded binary file (for PDFs, images, etc.).

Example request

curl

curl "https://scrape.shifter.io/v1?api_key=YOUR_API_KEY&url=https://httpbin.org/get"

Python

import requests

r = requests.get("https://scrape.shifter.io/v1", params={
    "api_key": "YOUR_API_KEY",
    "url": "https://httpbin.org/get",
})
print(r.text)

Node.js

import fetch from 'node-fetch';

const url = 'https://scrape.shifter.io/v1?' + new URLSearchParams({
  api_key: 'YOUR_API_KEY',
  url: 'https://httpbin.org/get',
});
const res = await fetch(url);
console.log(await res.text());

See also

• Basic request - minimal example
• Rendering JavaScript
• Proxies
• API errors