强制超时

有些页面在主要内容准备就绪后,仍有少数 DOM 节点会持续加载很长时间。使用 timeout 参数来限制总渲染时间。计时结束时,API 会返回当前已获取到的 HTML,若没有收集到任何可用内容,则返回 422 及相应的错误说明。

Endpoint

GET https://scrape.shifter.io/v1?api_key=YOUR_API_KEY&url=<TARGET_URL>&render_js=1&timeout=<MILLISECONDS>

Parameters

Parameter	Type	Required	Description
api_key	string	yes	您的 Web Scraping API 密钥
url	string	yes	要抓取的目标 URL
timeout	integer	no	引擎允许花费的最长渲染时间,单位为毫秒。最大值为 60000。
render_js	integer	no	设置为 1 以渲染 JavaScript。

Caution

非常短的超时时间(低于 1000 毫秒)通常会导致 422,因为页面尚未加载完成。建议从约 5000 毫秒开始,再逐步向下调整。

Example request

curl

curl "https://scrape.shifter.io/v1?api_key=YOUR_API_KEY&url=https%3A%2F%2Fhttpbin.org%2Fget&render_js=1&timeout=200"

Python

import requests

r = requests.get("https://scrape.shifter.io/v1", params={
    "api_key": "YOUR_API_KEY",
    "url": "https://httpbin.org/get",
    "render_js": 1,
    "timeout": 200,
})
print(r.json())

Node.js

import fetch from 'node-fetch';

const url = 'https://scrape.shifter.io/v1?' + new URLSearchParams({
  api_key: 'YOUR_API_KEY',
  url: 'https://httpbin.org/get',
  render_js: '1',
  timeout: '200',
});
const res = await fetch(url);
console.log(await res.json());

Example response

{
  "status": "Failure",
  "status_code": 422,
  "created_at": "2022-04-26T11:57:23.242Z",
  "processed_at": "2022-04-26T11:57:23.739Z",
  "time_taken": {
    "total": 0.701,
    "scraping": 0.202,
    "setup_worker": 0.403
  },
  "error": "The target page took more than 0.2 seconds to load, the website might be down. Retry the request or increase the value of 'timeout' parameter.",
  "page_content": null
}

See also

• 等待 CSS
• API 错误