Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
Shifter Now Has the Largest Live US Residential IP Pool
We re-ran Shifter through our residential proxy benchmark: 212,212 live US IPs in 250,000 requests, the largest live US pool of the 16 networks we have measured.
Elena Petrova · October 7, 2026 Who Owns Scraped Data? Rights, Licensing and What You Can Resell
Collecting web data does not make you its owner. Which rights sit in a scraped dataset, what you can license or resell, and what stays someone else's.
Sofia Marin · October 7, 2026 Geo-Grid Rank Tracking: Measuring Local Rankings Block by Block
A local business has no single Google rank: it changes every few blocks. How to measure it with a geo-grid, read the metrics, and act on the map.
Daniel Okafor · October 7, 2026 Web Data Collection for Academic Research: Reproducibility, Ethics Review and How to Cite a Scraped Dataset
Scraped data is now common in research, but often hard to reproduce, review or cite. How to collect web data that survives peer review, ethics boards and reuse.
James Meadow · October 6, 2026 Who Blocks the AI Crawlers? GPTBot, ClaudeBot and Friends Across the Top 10,000 Sites
We read the robots.txt of the top 10,000 domains. One site in five blocks at least one AI crawler, while almost all still welcome search engines.
Chris Collins · October 6, 2026 Decoding SERP API Pricing: What Drives Cost and How to Pay Less
A SERP API bill is keywords times depth times locations times devices times frequency. Why depth got ten times dearer in 2025, and how to pay less.
Matt Brown · October 5, 2026 Public-Record Collection for Investigative Reporting: Method, Ethics and Verification
Public records drive investigations, but only if collected carefully, preserved as served and checked before publication. A method for newsrooms.
James Meadow · October 5, 2026 The Cost of a Stale User-Agent: Measuring Browser Version Drift
Scrapers often hardcode a browser version and forget it. We requested 397 top sites claiming Chrome versions up to 6.5 years old. Refusals rose every year.
Chris Collins · October 5, 2026 Writing a Company Data Collection Policy: A Working Template
Most teams that collect web data have no written rules for it. A working policy template: scope, approvals, sources, personal data and takedowns.
Matt Brown · October 4, 2026