Every distributed crawler eventually has the same bad afternoon. A parser slows down because one site started shipping enormous pages. Fetchers keep fetching at full speed, because nothing told them not to. The queue between them grows by millions of items, memory climbs, a node falls over, its work is requeued onto the others, and the retries push them over too. Meanwhile the one site that caused it all is receiving more traffic than ever.
None of that is a bug in any single component. It is the absence of flow control between them. A crawler is a pipeline of stages with very different speeds, and without a way for a slow stage to tell a fast one to wait, the fast one wins until everything loses.
This essay is about building that feedback in: where the signals come from, where they need to go, and the handful of rules that make a crawler degrade gently instead of collapsing.
A crawler is a pipeline, and the stages disagree about speed
Strip away the details and most crawlers have four stages:
| Stage | What limits it | How it fails when overloaded |
|---|---|---|
| Frontier and scheduler | Almost nothing, it is just picking URLs | Emits work faster than anything downstream can absorb |
| Fetchers | Target sites, proxy capacity, plan limits | Timeouts, throttling, blocks |
| Parse and render | CPU, memory, headless browser slots | Growing latency, then memory exhaustion |
| Dedupe and storage | Database write capacity | Slow writes, lock contention, backlogs |
The frontier is nearly free to run, so it will always be the fastest stage. The other three are each limited by something different, and the limit moves: a site gets slower, a page type gets heavier, a database starts compacting. Flow control means that whichever stage is slowest right now sets the pace for all of them.
The alternative is that the pace is set by whichever stage runs out of memory first.
Rule 1: every queue is bounded
An unbounded queue between two stages is not a buffer. It is a promise to absorb any amount of excess work, which no machine can keep. It also hides the problem. The upstream stage sees success on every enqueue while the queue quietly grows, and by the time anyone looks, the oldest item has been waiting for hours.
A bounded queue turns the same situation into a signal. When it fills, the producer blocks, or gets an explicit rejection. The slowdown propagates upstream, one stage at a time, until it reaches the frontier, which simply stops handing out URLs for a while. Nothing is lost and nothing explodes.
In a single process this is one argument to a queue constructor:
import asyncio
async def fetcher(frontier: asyncio.Queue, parsed: asyncio.Queue, fetch):
while True:
url = await frontier.get()
try:
page = await fetch(url)
# Blocks when the parse queue is full: a slow parser slows fetching.
await parsed.put(page)
finally:
frontier.task_done()
async def parser(parsed: asyncio.Queue, store):
while True:
page = await parsed.get()
try:
await store(page)
finally:
parsed.task_done()
async def run(urls, fetch, store, fetchers=16, parsers=4):
frontier = asyncio.Queue(maxsize=1000)
parsed = asyncio.Queue(maxsize=200)
workers = [asyncio.create_task(fetcher(frontier, parsed, fetch)) for _ in range(fetchers)]
workers += [asyncio.create_task(parser(parsed, store)) for _ in range(parsers)]
for url in urls:
await frontier.put(url) # blocks when the frontier is full
await frontier.join()
await parsed.join()
for w in workers:
w.cancel()Across machines the principle is the same, only the mechanism changes: a broker queue with a length limit and a rejection policy, a stream with consumer lag you actually act on, or a database-backed frontier that only releases URLs to workers with free capacity. Size bounds from the latency you can tolerate, not from the memory you have. If the parse stage handles 200 pages a second and you accept 10 seconds of queueing, the queue holds about 2,000 items. Anything larger only postpones the moment you find out.
Rule 2: pull, don’t push
The cleanest backpressure is the kind you get for free. If workers ask for work when they have capacity, rather than a coordinator assigning work to them, a slow worker simply asks less often. The system cannot send it more than it can handle, because it never asked.
Pushing work makes the coordinator guess. It has to track each worker’s load, and it will guess wrong at exactly the moment the load changes. Pull-based designs, whether workers lease items from a queue or grant explicit credits to their upstream, move that decision to the only component that knows the answer.
Leases need expiry. A worker that dies holding work must not hold it forever, so leased items return to the queue after a timeout. Set that timeout from the slowest legitimate fetch, not the average, or healthy but slow work gets handed out twice.
Rule 3: queue per host, not one global queue
A single global fetch queue has a failure mode crawlers hit constantly: head-of-line blocking. If the next thousand URLs all belong to one slow or throttling site, every fetcher ends up waiting on that site while work for fast, healthy sites sits behind it.
Partition the fetch stage by host, or by whatever unit a target rate-limits on. Each host gets its own queue and its own concurrency limit, and fetchers pick from hosts that currently have room. One struggling site then slows only itself.
This is also where politeness lives. A per-host limit is simultaneously flow control for you and restraint toward the site. Setting that limit in the first place, and what the site’s own signals tell you about it, is covered in rate limiting and request throttling.
Rule 4: make per-host limits adaptive
A fixed per-host concurrency is wrong in both directions. Too low wastes capacity on a site that could take more. Too high keeps pressing on a site that has started pushing back, which is how temporary throttling becomes a block.
The well-tested answer is the one TCP uses for congestion: additive increase, multiplicative decrease. Every success nudges the limit up a little. Every throttle or timeout cuts it in half. The limit settles just below what the site will tolerate, and moves when the site changes.
import asyncio
class HostLimiter:
"""Adaptive concurrency for one host: additive increase, multiplicative decrease."""
def __init__(self, start=4, floor=1, ceiling=32):
self.limit = start
self.floor = floor
self.ceiling = ceiling
self.in_flight = 0
self._cond = asyncio.Condition()
async def acquire(self):
async with self._cond:
await self._cond.wait_for(lambda: self.in_flight < int(self.limit))
self.in_flight += 1
async def release(self, outcome):
async with self._cond:
self.in_flight -= 1
if outcome == "ok":
self.limit = min(self.ceiling, self.limit + 1 / max(1, int(self.limit)))
elif outcome in ("throttled", "timeout"):
self.limit = max(self.floor, self.limit / 2)
self._cond.notify_all()The increase is deliberately slow, roughly one extra slot per full round of successes, and the decrease is deliberately sharp. Asymmetry is the point: overshooting a site’s tolerance costs far more than undershooting it. Keep a hard ceiling per host that you set yourself, so an adaptive limit can never talk its way past what you consider reasonable.
Rule 5: know which signal came from where
Not every “slow down” means the same thing, and treating them alike sends the pressure to the wrong place.
| Signal | Where it originates | What it should slow |
|---|---|---|
429, rising latency, challenge pages from a site | The target | That host’s limiter only |
| Parse queue full, storage lagging | Your own pipeline | Fetching globally, then the frontier |
| Concurrency cap on a scraping API | Your plan | Total in-flight requests to that API |
| Quota or credits exhausted | Your plan | Everything, until the cycle resets or you top up |
Shifter’s Web Scraping API, for example, returns 429 Too Many Requests when you exceed your plan’s concurrency cap, and 509 when credits are exhausted. The first is a flow-control signal about you, not about any target, so it belongs in a global limiter on calls to the API, not in any per-host limiter. The second is a stop condition. Confusing either with a site’s own 429 makes a crawler throttle healthy sites for a problem that has nothing to do with them.
Retries are load
The most common way a crawler defeats its own backpressure is through retries. A fetch fails, the worker retries immediately, the retry fails too because the cause has not gone away, and every worker doing the same thing multiplies traffic at exactly the moment a target, or your own stage, is least able to take it.
- Retries pass through the same admission control as first attempts. A retry that skips the per-host limiter is a bypass around your own flow control.
- Back off with jitter, so a batch of workers that failed together does not retry together.
- Give each job a retry budget, and track retries as a share of total requests. When that share climbs, the system is spending its capacity on failure.
- Don’t stack retry layers. The Web Scraping API already retries failed fetches, CAPTCHAs and transient target errors up to three times with different proxies before returning an error, at no charge. Aggressive client-side retries on top of that multiply attempts rather than adding resilience.
- Never retry what cannot succeed. Authentication and configuration errors fail the same way every time.
When you cannot keep up, shed on purpose
Sometimes the honest answer is that the crawler has more work than it can do. Backpressure then slows everything evenly, which is often the worst outcome: every job finishes late, including the ones that matter.
Load shedding makes the choice explicit. Give work a priority, and when queues stay full past a threshold, drop or defer the lowest-priority items at the frontier, before they cost a fetch. Refreshing a volatile price page is worth more than re-checking an archive page that has not changed in a year. Which pages deserve the budget is its own problem, and the subject of cost-aware crawl scheduling.
Measure the pressure, not just the throughput
Throughput graphs look fine right up until the collapse. The metrics that show pressure building are different:
- Queue depth and the age of the oldest item, per stage. Age matters more than depth: a deep queue that drains quickly is healthy, a shallow one full of stale items is not.
- In-flight requests and the current adaptive limit, per host. A limit that keeps halving is a site pushing back.
- Admission rejections and blocked puts, per stage. This is backpressure doing its job. A sudden rise tells you which stage is now the bottleneck.
- Retry share, per host and overall.
A falling per-host limit combined with rising block and challenge rates usually means a site is turning against you rather than just being slow. Turning those signals into a single judgement per site is covered in building a target health score. The wider set of pipeline metrics is in monitoring a web scraping pipeline.
Scaling out doesn’t remove the need for flow control
Adding workers raises the ceiling of the fetch stage. It does not raise any target’s tolerance, your parse capacity or your plan’s limits. A crawler that scales fetchers automatically on queue depth, without per-host limits and bounded downstream queues, will scale straight into every constraint at once. If you run on Kubernetes, scale on the signals above rather than on CPU alone; the mechanics are covered in scaling residential proxy scraping with Kubernetes.
The same applies across regions. Durable queues and backpressure between regions are what keep a regional failure from turning into a global one, as described in residential proxy failover for multi-region pipelines.
The bottom line
Flow control is not an optimisation to add once a crawler is large. It is what decides whether a crawler under stress slows down or falls over. The rules are few: bound every queue, let workers pull, partition by host, adapt per-host limits sharply downward and slowly upward, route each slowdown signal to the stage it describes, treat retries as load, and shed the least valuable work deliberately.
A crawler built this way has one more property worth having. When a site starts to struggle, the crawler notices and eases off on its own, which is good for the site and, not by coincidence, good for the crawler’s odds of still being welcome next week.