Desktop crawlers such as Screaming Frog and Sitebulb send hundreds or thousands of requests to one site in a short time, all from one IP address. On your own site that is rarely a problem. On a competitor’s site, a large ecommerce catalogue or anything behind a strict firewall, the crawl often stops halfway: 403 and 429 responses, CAPTCHA pages counted as content, or a site that quietly serves a different version to your office IP.
A proxy fixes the address side of that. It is also how you audit a site the way visitors in another country see it, which matters for international SEO. This guide covers when an audit actually needs a proxy, how to set one up in the main tools, and the settings that decide whether the crawl works. It is part of our wider SEO monitoring work, where seeing pages from the right place is half the job.
When a site audit needs a proxy
Auditing sites you do not control. Competitor audits, content gap research and pre-sale audits for prospects all crawl sites that never agreed to your crawl. Firewalls and CDNs rate-limit by IP, and a burst from one address is the easiest pattern for them to block.
Checking localised versions. Sites redirect by country, change prices and currency, and sometimes hide whole sections outside a market. A crawl from your office only ever sees one version. Crawling through an IP in each target country shows what visitors and local search users get, and lets you check whether hreflang alternates actually resolve. Our guide to using hreflang to find every language version of a page covers that side of the audit.
Finding geo or network-based tricks. Some sites, and many hacked ones, serve different content to office and datacenter networks than to home users. A crawl from home connections in the right market is how you see the real version.
When you do not need one. For your own site, the better fix is to allowlist your crawler’s IP address in your firewall or CDN, or to crawl a staging copy. A proxy adds cost and another moving part to a crawl you can simply permit.
Setting up a proxy in Screaming Frog
Screaming Frog’s SEO Spider has a proxy setting under File > Settings > Proxy on Windows and Linux, and under Screaming Frog SEO Spider > Settings > Proxy on macOS. Tick “Use Proxy Server”, enter the proxy address and port, and supply your proxy credentials.
Three details from Screaming Frog’s own documentation shape how you use it:
- Only one proxy server can be configured. There is no built-in list rotation, so the proxy itself has to handle rotation if you want it.
- You must restart the tool for proxy changes to take effect.
- There are no exceptions. Either all HTTP and HTTPS traffic from the crawler goes through the proxy, or none of it does.
The one-proxy limit is why a gateway-style residential proxy suits crawlers well: you enter a single endpoint, and the gateway decides which exit IP each request uses, based on the targeting in your username. If the crawl fails straight away with authentication errors, our guide to fixing 407 proxy authentication errors lists the usual causes.
Setting up a proxy in Sitebulb
In Sitebulb, open Settings from the top navigation and go to the Proxy tab. These are global settings, so they become the default for every new audit you start. Remember to switch the proxy off again before auditing your own sites, or those audits will run through it too.
Other audit tools follow the same pattern: a single proxy field in the application or project settings. Cloud-based crawlers usually run from their own servers and do not accept a proxy at all, which is one reason desktop crawlers remain common for competitor work.
Rotating or sticky: choosing the session type
With a gateway, you choose whether each request gets a new exit IP or whether the crawl keeps one IP for a while.
Rotating per request spreads the crawl across many addresses, so no single IP sends enough traffic to trip a rate limit. It works well on static pages and large catalogues with no logins and few cookies.
Sticky sessions keep the same IP for a set time. Use them when the site ties cookies, consent choices or a chosen country to the session, or when switching IP mid-crawl triggers fresh bot checks. Set the session time longer than a typical burst of the crawl so the IP does not change while related pages load.
Our decision framework for rotating proxy IPs goes through the trade-off in more detail. For localised audits, add the country, and if needed the city, to the proxy username so every request exits in the market you are auditing.
Crawl settings that matter more than the proxy
A proxy changes where requests come from. It does not make an aggressive crawl acceptable, and several settings in the crawler matter just as much.
Speed. Screaming Frog crawls with 5 threads by default and lets you cap the crawl in URLs per second under Configuration > Speed. On someone else’s site, keep the rate low and let the crawl take longer. Spreading a fast crawl across many IPs is still a fast crawl for the server receiving it.
User agent. Under Configuration > User-Agent you can crawl as a browser or as Googlebot. A Googlebot user agent from a non-Google IP is easy to spot, and sites that verify crawlers will treat it as suspicious. For competitor audits, a normal browser user agent is the safer choice.
What you fetch. Residential traffic is usually billed per GB. Images, fonts, video and scripts can be most of a page’s weight. If the audit is about HTML, titles, links and status codes, exclude the resource types you do not need. JavaScript rendering loads every resource a browser would, so enable it only for the sections that need it.
Robots.txt. Respecting robots.txt is the default in both tools. Keep it that way for sites you do not own.
FAQ
Can Screaming Frog rotate proxies?
Not from a list. Screaming Frog accepts a single proxy server. To rotate IPs, point it at a proxy gateway that assigns a new exit IP per request or per session.
Do I need a proxy to crawl my own website?
Usually not. Allowlisting your crawler’s IP in your firewall or CDN is simpler and cheaper. Use a proxy on your own site only when you need to see it from another country or network.
Which proxy type works best for SEO crawlers?
Residential proxies are the most reliable for sites with strict bot protection and for seeing localised versions, because requests come from home connections in the chosen country. Datacenter proxies are cheaper and can work on sites with little protection.
Why does my crawl show 403 or 429 errors?
The site is blocking or rate-limiting your requests. Lower the crawl speed first, then crawl through rotating IPs if the limits are per address.
The bottom line
Desktop crawlers send every request from one IP, which is fine for your own site and a problem for everyone else’s. For competitor and international audits, point Screaming Frog or Sitebulb at a single gateway endpoint, choose rotating or sticky sessions to match the site, target the country you are auditing, and keep the crawl slow and focused on the pages and resources you actually need.