Forcing timeouts

Some pages keep a few DOM nodes loading long after the main content is ready. Use the timeout parameter to cap total render time. The API returns whatever HTML it has when the timer fires, or a 422 with an explanatory error if nothing usable was collected.

Endpoint

GET https://scrape.shifter.io/v1?api_key=YOUR_API_KEY&url=<TARGET_URL>&render_js=1&timeout=<MILLISECONDS>

Parameters

Parameter	Type	Required	Description
api_key	string	yes	Your Web Scraping API key
url	string	yes	Target URL to fetch
timeout	integer	no	Maximum time the engine may spend rendering, in milliseconds. Max 60000.
render_js	integer	no	Set to 1 to render JavaScript.

Caution

Very short timeouts (under 1000 ms) often yield a 422 because the page has not finished loading. Start around 5000 ms and tune down from there.

Example request

curl

curl "https://scrape.shifter.io/v1?api_key=YOUR_API_KEY&url=https%3A%2F%2Fhttpbin.org%2Fget&render_js=1&timeout=200"

Python

import requests

r = requests.get("https://scrape.shifter.io/v1", params={
    "api_key": "YOUR_API_KEY",
    "url": "https://httpbin.org/get",
    "render_js": 1,
    "timeout": 200,
})
print(r.json())

Node.js

import fetch from 'node-fetch';

const url = 'https://scrape.shifter.io/v1?' + new URLSearchParams({
  api_key: 'YOUR_API_KEY',
  url: 'https://httpbin.org/get',
  render_js: '1',
  timeout: '200',
});
const res = await fetch(url);
console.log(await res.json());

Example response

{
  "status": "Failure",
  "status_code": 422,
  "created_at": "2022-04-26T11:57:23.242Z",
  "processed_at": "2022-04-26T11:57:23.739Z",
  "time_taken": {
    "total": 0.701,
    "scraping": 0.202,
    "setup_worker": 0.403
  },
  "error": "The target page took more than 0.2 seconds to load, the website might be down. Retry the request or increase the value of 'timeout' parameter.",
  "page_content": null
}

See also

• Waiting for CSS
• API errors