Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
LLM Extraction vs Selectors: When to Let a Model Read the Page
We had a large language model extract fields from 18 live news pages and scored it against the pages' own structured data. Accuracy, cost, and when to use it.
Chris Collins · September 28, 2026 Change Detection at Scale: Diffing Web Pages Without Drowning in Noise
Pages change on every load: ads, timestamps, tokens. How to tell real changes from noise with normalisation, field-level diffs and simhash, with tested code.
Chris Collins · September 27, 2026 Stop Parsing HTML: Extracting Structured Data From JSON-LD and Embedded Data
Most pages already carry machine-readable data. How to extract JSON-LD and embedded JSON instead of brittle selectors, and what to do when it is incomplete.
Matt Brown · September 26, 2026 Building a Target Health Score: Knowing a Site Is Turning Against You
Sites rarely block a crawler in one step. They challenge, soft-block and slow down first. How to turn those signals into one score per site, and act on it.
Matt Brown · September 22, 2026 Cost-Aware Crawl Scheduling: Spending Your Budget Where It Changes
Most recrawl budget is spent confirming nothing changed. How to schedule by expected value per cost, and why the pages that change fastest are not the best buy.
James Meadow · September 22, 2026 Backpressure and Flow Control in Distributed Crawlers
A crawler is a pipeline, and its fastest stage will flood its slowest. How bounded queues, pull-based work and adaptive per-host limits keep a crawler stable.
Chris Collins · September 22, 2026 Running Scrapers in CI/CD: Proxy Credentials, Secrets and Test Runs
How to run scrapers in CI without leaking proxy credentials or burning bandwidth on every push: secret storage, test tiers, sessions, quota checks and rotation.
Chris Collins · September 21, 2026 How to Monitor Employer Reviews and Ratings Across Sites
Employer reviews shape who applies. How to monitor ratings and themes across review sites, benchmark competitors, and stay within each site's rules.
James Meadow · September 19, 2026 How to Build Large-Scale Training Datasets from the Open Web
Collecting pages is the easy part. How to turn an open-web crawl into a training corpus: frontier, opt-outs, extraction, filtering, dedup and decontamination.
Matt Brown · September 18, 2026