Guides, Tutorials & Insights
Learn about residential proxies, web scraping, data collection strategies, and industry best practices from the Shifter team.
How to Use Residential Proxies in Puppeteer (Including page.authenticate)
Proxies in Puppeteer: the proxy goes in launch args, credentials in page.authenticate, per-page geo rotation through one gateway, and resource blocking.
Chris Collins · August 5, 2026 The State of Web Scraping in 2026
AI made data collection strategic, anti-bot went network-level, blocks turned invisible, pricing shifted to bandwidth. Where web scraping stands in 2026.
James Meadow · August 3, 2026 How to Use Residential Proxies in C# with HttpClient
Proxies in C#: WebProxy with NetworkCredential, SocketsHttpHandler and PooledConnectionLifetime, IHttpClientFactory, and per-identity clients for geo rotation.
Chris Collins · August 2, 2026 Build vs Buy: Should You Run Your Own Scraping Infrastructure?
A demo scraper takes an afternoon. Running it at scale is a product. A framework for what to build, what to buy, and the one layer to always rent.
Chris Collins · August 1, 2026 How to Use Residential Proxies in Node.js with Axios and Got
Proxies in Node.js: why Axios needs https-proxy-agent and proxy:false, Got's agent option, undici's ProxyAgent for fetch, and per-request geo rotation.
Chris Collins · July 29, 2026 How to Scale Residential Proxy Scraping Workloads with Kubernetes
Run scraping on Kubernetes: proxy creds as Secrets, per-pod session identity via the Downward API, queue-driven autoscaling, and graceful shutdown.
Chris Collins · July 28, 2026 How to Use Residential Proxies in PHP with cURL and Guzzle
Proxies in PHP: cURL's CURLOPT_PROXY and CURLOPT_PROXYUSERPWD, Guzzle's proxy option, CURLOPT_HTTPPROXYTUNNEL, and per-request geo rotation.
Chris Collins · July 26, 2026 How to Use Residential Proxies in Java with OkHttp and Apache HttpClient
Proxies in Java: OkHttp's proxyAuthenticator gotcha, Apache HttpClient's default-5 per-route pool limit, entity consumption, and per-request geo rotation.
Chris Collins · July 25, 2026 How to Use Residential Proxies in Go with net/http and Colly
Proxies in Go: transport setup, the body-drain and MaxIdleConnsPerHost gotchas that kill connection reuse, per-request geo rotation, and Colly's SetProxyFunc.
Chris Collins · July 24, 2026