Look at the fetch logs of almost any long-running crawler and the same pattern appears. The large majority of requests return a page identical to the last copy. Each of those fetches cost bandwidth or credits, occupied a concurrency slot and put load on someone else’s server, and produced exactly nothing.
That is not a bug in the fetcher. It is a scheduling decision, usually made by default: revisit everything on the same cycle, or revisit the important things more often. Both feel reasonable, and both waste most of the budget. This essay is about making the decision on purpose, pricing each visit by what it is likely to buy.
Measure the right unit
Teams usually track cost per page fetched. The number that matters is cost per change detected.
A crawler that fetches a million pages a day and detects ten thousand changes has spent a hundred fetches per useful observation. Halving the fetch count without losing changes halves the cost of the dataset. Doubling the fetch count without finding more changes doubles it for nothing. Put “fetches that found no change” on a dashboard, per site and per page type, and it becomes obvious where the money goes.
Know what a fetch actually costs
The cost of a visit depends on how you are billed, and the two common models reward different optimisations.
| Billing model | What you pay for | What makes a visit cheaper |
|---|---|---|
| Residential proxy bandwidth | Bytes through the gateway | Smaller responses: no rendering, no images, compressed transfer |
| Scraping API credits | Successful responses | Fewer fetches; the size of each matters much less |
On Shifter’s residential gateway every byte sent or received counts, including headers, and failed requests count if bytes were transferred before the error. Shrinking each fetch pays off directly, and the tactics for that are covered in cutting proxy bandwidth costs. On the Web Scraping API a successful request costs one credit whether or not JavaScript rendering is enabled, and failed requests and target errors cost nothing. There, a rendered page and a plain one cost the same, and the only lever that moves the bill is how many successful fetches you ask for.
Either way the scheduler is the biggest lever you have, because the cheapest fetch is the one you decided not to make.
How often does each page change?
Scheduling by expected value needs an estimate of how often each page changes. The standard working assumption in crawler research is that changes arrive roughly at random at some average rate per page, which makes the probability a page has changed since your last visit:
P(changed) = 1 - e^(-rate × time since last visit)
You estimate the rate from your own history. Each visit tells you whether the page changed since the one before. Divide changes seen by time observed, per page, and smooth it toward the average for similar pages so that a page visited three times does not get an extreme estimate.
Two cautions. First, a visit can only tell you that a page changed, not how many times, so pages that change faster than you visit look slower than they are. Second, define “changed” by the fields you care about. A page whose timestamp, ad slot or session token differs on every load changes constantly and means nothing. Hash the extracted record, not the HTML.
The counterintuitive result: don’t chase the fastest pages
The obvious policy is to visit pages in proportion to how often they change. It is also wrong, and the proof is more than twenty years old.
In “Effective Page Refresh Policies for Web Crawlers” (ACM Transactions on Database Systems, 2003), Junghoo Cho and Hector Garcia-Molina compared allocation policies for keeping a local copy fresh with a fixed visit budget. Their finding, in their words, is that “the uniform policy is always more effective than the proportional policy under any scenario”. Their optimal policy goes further, and they summarise it directly: “To improve freshness, we should penalize the elements that change too often.”
The intuition is simple once stated. A page that changes every fifteen minutes is stale again almost as soon as you fetch it. Visiting it buys a few minutes of freshness. A page that changes about once a day, visited daily, stays correct for most of the day after each visit. With a limited budget, the second is a far better purchase.
You can put a number on it. Under the same change model, the freshness a visit buys is the chance the page is currently stale multiplied by how long it is likely to stay correct afterwards. For a crawler that can revisit about once a day:
| Page changes | Freshness bought per visit (days) |
|---|---|
| Every 15 minutes or so | 0.010 |
| About hourly | 0.042 |
| Every 6 hours or so | 0.241 |
| About daily | 0.400 |
| About weekly | 0.124 |
| About monthly | 0.032 |
| About yearly | 0.003 |
The best buys are pages that change at roughly the pace you can afford to revisit. Pages that change much faster are nearly impossible to keep fresh at any affordable rate; pages that change much slower are nearly always fresh already.
There is one important exception. The result is about freshness: how often your copy matches the live page. Some jobs are about capturing events instead: every price change, every stock-out, every edit. If each change matters individually, fast-changing pages need more visits, not fewer, and the right answer may be a different source entirely, such as an API, a feed or a listing page that shows the change without a full fetch. Decide which problem you are solving before you tune.
Score every visit by expected value per cost
Putting the pieces together, each candidate visit gets a priority: how much the page matters, times the freshness a visit would buy, divided by what the visit costs.
import math
def freshness_gain(rate, days_since_visit, interval_days):
"""Expected fresh days bought by visiting now, under a Poisson change model."""
p_stale = 1 - math.exp(-rate * days_since_visit)
fresh_after = (1 - math.exp(-rate * interval_days)) / rate if rate > 0 else interval_days
return p_stale * fresh_after
def priority(page, today, interval_days=1.0):
gain = freshness_gain(page.change_rate, today - page.last_fetched, interval_days)
return page.value * gain / page.costThe scheduler then works from a priority queue: each cycle, spend the budget on the highest-scoring visits and stop. Three inputs deserve care.
- Value is a business judgement, not a technical one. A product that sells, a competitor that matters, a query your customers run. Keep it coarse; three or four tiers are usually enough.
- Cost should be the real cost of that visit: bytes for that page type, credits, and whether it needs rendering. A page that needs a headless browser on a bandwidth-billed plan can cost many times a plain fetch.
- Change rate comes from your history, updated after every visit.
Cheap signals before expensive fetches
Often you can learn whether a page changed for much less than the price of fetching it.
- Conditional requests. Where a site honours
ETagorLast-Modified, a304 Not Modifiedresponse costs a fraction of the bytes. Track per host whether validators are reliable; some sites send them and ignore them. - Listing pages as change detectors. A category or search page often shows price and availability for dozens of items. Fetch the listing, compare, and fetch only the items whose summary changed. This is how most real-time price feeds and stock monitors stay affordable.
- Sitemaps and feeds. Where they carry trustworthy modification dates, they tell you what changed without visiting anything else.
- Structured endpoints. A JSON response behind a page is usually smaller and more stable than the page itself.
Reserve budget for what the scheduler cannot see
A scheduler that only optimises known pages will slowly go blind. Hold back part of every cycle for three things:
- Discovery. New URLs have no history and would never outrank established pages. Give them their own allocation.
- Re-estimation. A page scored as unchanging for months should still be visited occasionally, because pages change behaviour. Without this, a wrong estimate never gets corrected.
- Verification. A small random sample, fetched regardless of score, tells you whether the model’s assumptions still hold.
Let cost and health push back on the schedule
The schedule is a plan, not a guarantee. When a site starts throttling, the scheduler should hear about it and re-rank, rather than keep queueing visits the fetch stage cannot make. That feedback path, from fetchers back to the frontier, is covered in backpressure and flow control in distributed crawlers, and a site’s overall condition in building a target health score. A falling health score should also raise the effective cost of visiting that site, which is exactly the signal a cost-aware scheduler needs.
The bottom line
A crawl budget spent evenly, or in proportion to how busy each page is, is mostly spent confirming that nothing happened. Estimate how often each page changes from your own history, price each visit by what it will buy and what it will cost, and give the budget to the best buys first. Expect those to be the pages that change at about the rate you can afford to follow, not the ones that change fastest.
The payoff is not only a smaller bill. A crawler that fetches less, and fetches more carefully, is also lighter on the sites it depends on.
Sources and references
- Junghoo Cho and Hector Garcia-Molina, Effective Page Refresh Policies for Web Crawlers, ACM Transactions on Database Systems, Vol. 28, No. 4, December 2003.
- Shifter, Residential Proxies bandwidth and billing. What counts as traffic.
- Shifter, Web Scraping API errors and limits. Credit costs, failure behaviour and retries.