Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
Who Blocks the AI Crawlers? GPTBot, ClaudeBot and Friends Across the Top 10,000 Sites
We read the robots.txt of the top 10,000 domains. One site in five blocks at least one AI crawler, while almost all still welcome search engines.
Chris Collins · October 6, 2026 The Cost of a Stale User-Agent: Measuring Browser Version Drift
Scrapers often hardcode a browser version and forget it. We requested 397 top sites claiming Chrome versions up to 6.5 years old. Refusals rose every year.
Chris Collins · October 5, 2026 Is Your Target Blockable? An Anti-Bot Stack Lookup
Before building a scraper, find out what protects the target. We checked the top 1,000 domains: which anti-bot vendors they use and how a plain client fared.
Chris Collins · October 3, 2026 Proxy Selection as a Multi-Armed Bandit Problem
Which proxy configuration should the next request use? Treat it as a bandit problem. In our simulation, adaptive selection came within 2% of perfect hindsight.
James Meadow · October 2, 2026 Designing Idempotent Scrapers: Restarting Without Duplicating
Scrapers crash. We killed one ten times mid-crawl: a naive version stored up to 4.3 times the rows, an idempotent one exactly 500. How to build the second.
Chris Collins · October 2, 2026 Character Encodings in Scraped Data: Detecting Charsets and Fixing Mojibake
We fetched 193 top homepages in six countries. A common Python default garbled the text of 37 of them. How to decode scraped pages the way browsers do.
Chris Collins · October 1, 2026 Keep the Raw Response: Archiving Scraped Pages in WARC for Replay
Parsers improve and extractors break, but a page you did not keep cannot be parsed again. How to store raw responses in WARC, with tested code and real sizes.
Matt Brown · October 1, 2026 Finding the API Behind the Page: Collecting From JSON Endpoints Instead of HTML
Many pages load their data as JSON. How to find the endpoint that carries it, why it is often tied to the page's session, and how to collect it responsibly.
Chris Collins · September 30, 2026 Sitemaps as a Discovery Source: Finding Every URL Without Crawling the Whole Site
XML sitemaps list a site's URLs, and sometimes when they changed. How to read them properly, and why you should never trust lastmod without checking it.
James Meadow · September 29, 2026