Retry Economics: The Attempt Number Where Retrying Stops Paying for Itself

Every scraper retries failed requests, and most pick the number of retries the same way: three, because it feels sensible. But retries are not free. On residential proxies billed by the gigabyte, every retry downloads the page again; on any system, it costs time and load on the target. And each retry is usually less likely to succeed than the one before.

So there is a point where another attempt costs more than it is worth. This study shows how to find it from your own logs, with a worked example and the code to run on your data.

Why later retries succeed less often

If failures were pure bad luck, every retry would have the same chance of success as the first try. In practice they are not. A request that fails once is more likely to be failing for a reason that persists: a page that does not exist, a block on that address, a site that is down, a parser that never matches. The jobs that reach attempt four are mostly the hard cases.

That means the success rate per attempt falls, and the cost of each extra success rises with it.

Measure it from your logs

The only data you need is, for each job, the attempt number that succeeded, or a note that every attempt failed. Most scrapers already log this. From it, the success rate of attempt k is the share of jobs that reached attempt k and succeeded there.

from collections import Counter


def attempt_success_rates(outcomes, max_attempts):
    """Chance that attempt k succeeds, given the job got that far.

    outcomes: per job, the attempt number that succeeded, or None if every
    attempt failed (with max_attempts tries).
    """
    succeeded_at = Counter(o for o in outcomes if o is not None)
    reached = len(outcomes)
    rates = []
    for k in range(1, max_attempts + 1):
        rates.append(succeeded_at[k] / reached if reached else 0.0)
        reached -= succeeded_at[k]
    return rates


def cost_per_success(rates, cap, attempt_cost):
    """Expected cost of one successful job if we stop after `cap` attempts."""
    p_reach, spent, success = 1.0, 0.0, 0.0
    for rate in rates[:cap]:
        spent += p_reach * attempt_cost
        success += p_reach * rate
        p_reach *= 1 - rate
    return spent / success if success else float("inf")


def best_cap(rates, attempt_cost, value_of_success):
    """The last attempt whose expected value still exceeds its cost."""
    cap = 0
    for k, rate in enumerate(rates, start=1):
        if rate * value_of_success > attempt_cost:
            cap = k
    return cap

Three functions:

• attempt_success_rates turns your log into a success rate per attempt.
• cost_per_success gives the average cost of one successful job if you stop after a given number of attempts.
• best_cap finds the last attempt whose expected value is still higher than its cost.

A worked example

Here is an illustrative log of 10,000 jobs, each allowed six attempts:

from retry_econ import attempt_success_rates, best_cap, cost_per_success

# Illustrative outcomes for 10,000 jobs, each allowed up to six attempts:
# the attempt that succeeded, or None if all six failed.
outcomes = [1] * 8200 + [2] * 900 + [3] * 260 + [4] * 90 + [5] * 40 + [6] * 20 + [None] * 490
rates = attempt_success_rates(outcomes, max_attempts=6)
print("success rate by attempt:", [round(r, 3) for r in rates])

for cap in range(1, 7):
    print(f"cap {cap}: cost per success {cost_per_success(rates, cap, attempt_cost=1.0):.3f}")

print("best cap if a success is worth 5 attempts:", best_cap(rates, attempt_cost=1.0, value_of_success=5.0))
print("best cap if a success is worth 20 attempts:", best_cap(rates, attempt_cost=1.0, value_of_success=20.0))

success rate by attempt: [0.82, 0.5, 0.289, 0.141, 0.073, 0.039]
cap 1: cost per success 1.220
cap 2: cost per success 1.297
cap 3: cost per success 1.357
cap 4: cost per success 1.412
cap 5: cost per success 1.464
cap 6: cost per success 1.514
best cap if a success is worth 5 attempts: 3
best cap if a success is worth 20 attempts: 5

Three things stand out.

The success rate falls fast. The first attempt succeeds 82% of the time. Of the jobs that reach attempt two, half succeed. By attempt five, fewer than one in ten do.

Each extra success costs more than the last. Dividing the cost of one attempt by its success rate gives the cost of a success from that attempt: about 1.2 attempts’ worth on the first try, 2 on the second, 3.5 on the third, 7 on the fourth and over 25 on the sixth. The average cost per success rises slowly as you allow more attempts, because each extra attempt runs on fewer jobs, but the marginal cost climbs steeply.

The right cap depends on what a success is worth. If a successful record is worth five attempts’ cost, stop after three. If it is worth twenty, as it might be for a page you must have, go to five. There is no single right number; there is a right number for each value.

Putting a value on a success

The value of a success is the one input you have to decide. A few ways to set it:

• Replaceable records, such as one price observation among many collected daily, are worth little each. Missing one is fine; a low cap is right.
• Required records, such as a page that completes a dataset a customer is waiting for, are worth much more. A higher cap makes sense.
• Costs beyond bandwidth count too: time, load on the target and the chance that hammering a failing page draws a block for everything else.

On a web scraping API that charges only for successful responses, the money side of a retry shifts to the provider, but time and target load still apply. Shifter’s Web Scraping API, for example, does not charge for failed requests.

Making retries cheaper

The cap is only half of it. The other half is making each retry more likely to succeed:

• Wait before retrying, and wait longer each time. Many failures are temporary rate limits that clear on their own.
• Change something when the failure suggests it. A block on one address may succeed from another; a timeout may need a longer timeout, not the same request again.
• Do not retry what cannot succeed. A 404 or an authentication error will fail every time.

Our guide to retry and backoff for web scrapers covers these techniques in detail.

Run it on your own data

1. Export, for a week of jobs, the attempt that succeeded or a failure marker.
2. Run attempt_success_rates with your current maximum.
3. Decide what a success is worth in attempts.
4. Set the cap to what best_cap returns, per job type if they differ.
5. Re-check monthly. Rates change as sites and your setup change.

FAQ

How many times should a scraper retry a failed request?

It depends on how fast your per-attempt success rate falls and what a successful record is worth. Measure the success rate of each attempt from your logs, then stop at the last attempt whose expected value is still higher than its cost. For many jobs that is two to four attempts.

Why do retries succeed less often than first attempts?

Because failures are not random. Jobs that fail once are more likely to have a persistent problem, such as a missing page or a block, so each later attempt is drawn from harder cases.

Do retries cost money with residential proxies?

Yes. Residential proxy traffic is billed by volume, so every retry downloads the page again. Failed attempts that return large error pages cost bandwidth too.

Should I retry 404 errors?

No. A page that does not exist will not appear on the next attempt. Retry only failures that can succeed later, such as timeouts, rate limits and temporary server errors.

The bottom line

Retrying is worth it until the chance that the next attempt succeeds, times what a success is worth, drops below the cost of trying. Your logs already contain the success rate of every attempt; measure it, put a value on a success, and set the cap from the numbers instead of from habit.