Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
Normalising Prices, Numbers and Dates Across Locales
1.234,56 € and $1,234.56 are both prices, and 03/04/2026 is two different dates. How to parse scraped values correctly by locale, with tested code.
James Meadow · September 30, 2026 Finding the API Behind the Page: Collecting From JSON Endpoints Instead of HTML
Many pages load their data as JSON. How to find the endpoint that carries it, why it is often tied to the page's session, and how to collect it responsibly.
Chris Collins · September 30, 2026 Introducing IP Info: A Free IP Geolocation API, No API Key Needed
IP Info is Shifter's free IP geolocation and ASN API. It runs on the same database behind our proxy pools, needs no signup or API key, and is ready for AI agents.
James Meadow · September 29, 2026 Schema Drift in Scraped Data: Catching Broken Extractors Before Your Users Do
Scrapers rarely crash when a site changes. They keep running and return subtly wrong data. How data contracts and batch profiling catch drift early.
Matt Brown · September 29, 2026 Sitemaps as a Discovery Source: Finding Every URL Without Crawling the Whole Site
XML sitemaps list a site's URLs, and sometimes when they changed. How to read them properly, and why you should never trust lastmod without checking it.
James Meadow · September 29, 2026 LLM Extraction vs Selectors: When to Let a Model Read the Page
We had a large language model extract fields from 18 live news pages and scored it against the pages' own structured data. Accuracy, cost, and when to use it.
Chris Collins · September 28, 2026 Deduplicating Scraped Data: Entity Resolution for Products, Companies and Listings
The same product, company or listing shows up many times across sources and runs. How to normalise identifiers, match safely at scale and keep one clean record.
Matt Brown · September 28, 2026 Cost per Clean Record: The Scraping Metric Finance Actually Understands
Gigabytes and request counts do not tell finance what data costs. How to measure cost per clean record and per useful change, and which levers move it most.
James Meadow · September 27, 2026 Change Detection at Scale: Diffing Web Pages Without Drowning in Noise
Pages change on every load: ads, timestamps, tokens. How to tell real changes from noise with normalisation, field-level diffs and simhash, with tested code.
Chris Collins · September 27, 2026