Every scraper eventually stops halfway. A container is rescheduled, a deployment restarts the worker, a machine runs out of memory, or someone presses Ctrl+C. What happens next decides whether your data stays trustworthy. A scraper that simply starts again from the top re-fetches everything and, unless it was designed for it, writes everything again.
An idempotent scraper is one where running the same work twice has the same effect as running it once. It can be killed at any moment, restarted, and still end up with exactly one correct copy of each record. This guide shows the four design decisions that make that true, with tested code and the results of killing it ten times mid-crawl.
Key takeaways
- Assume the process will die between any two lines. Design so that a restart repeats at most the one page in flight.
- Give every record an ID derived from what it is, such as the source and SKU, never from when it was scraped. Then a repeat write is an update, not a duplicate.
- Keep the crawl frontier in durable storage, and mark a URL done in the same transaction that stores its result.
- Use leases rather than locks, so work claimed by a crashed worker becomes available again on its own.
- In our test of 500 products, killed ten times per run: the naive scraper ended with 1,286 to 2,165 rows and up to 4.3 times the requests; the idempotent one ended with exactly 500 correct rows and at most 3 repeated requests.
Why restarts create duplicates
The common first version of a scraper looks like this: start at page one, follow the pagination, insert a row for every product, commit as you go. It works perfectly until it is interrupted. On restart it has no memory of how far it got, so it starts again, and every product it already stored is inserted a second time with a new auto-incremented ID. Kill it a few times and the table holds several copies of most products, with nothing to say which one is current.
Deduplicating afterwards is possible, and entity resolution covers it, but it treats the symptom. The duplicates also cost money before they cost accuracy: every repeated page is bandwidth paid for twice, which feeds straight into cost per clean record.
Four decisions that make a scraper idempotent
1. Derive record IDs from the data
A record’s ID should come from what the record is: the source plus its natural key, such as a SKU, a listing ID or a canonical URL. Hash them together and the same product always gets the same ID, on every run, on every machine. Writing it again becomes an upsert: if the record exists and is unchanged, nothing happens; if it changed, it is updated and the change time recorded.
2. Keep the frontier durable
The list of URLs to visit, and which ones are done, belongs in the database, not in memory. Adding a URL that is already known must be a no-op, so rediscovering links on a listing page you have already processed is harmless.
3. Commit the result and the progress together
The dangerous moment is between storing a result and recording that the URL is finished. If the process dies after one and before the other, a restart either loses the result or repeats it. Doing both in one database transaction removes the gap: either the record is stored and the URL marked done, or neither happened and the URL is simply fetched again.
4. Lease work instead of locking it
A worker claims a URL for a limited time. If it finishes, the URL is marked done. If it crashes, the lease expires and another worker, or the restarted one, picks the URL up. Nothing has to notice the crash for the work to be recovered.
The code
The module below implements all four with SQLite and requests. SQLite keeps the example self-contained; the same design carries over to PostgreSQL or any database with transactions and upserts.
import hashlib
import json
import re
import sqlite3
import time
import requests
SCHEMA = """
CREATE TABLE IF NOT EXISTS frontier (
url TEXT PRIMARY KEY,
status TEXT NOT NULL DEFAULT 'pending', -- pending, leased, done
leased_until REAL NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS records (
record_id TEXT PRIMARY KEY, -- derived from the source and its natural key
url TEXT NOT NULL,
content_hash TEXT NOT NULL,
data TEXT NOT NULL,
first_seen REAL NOT NULL,
last_changed REAL NOT NULL
);
"""
def open_db(path):
db = sqlite3.connect(path, isolation_level=None) # explicit transactions below
db.execute("PRAGMA journal_mode=WAL")
db.executescript(SCHEMA)
return db
def record_id(source, natural_key):
"""The same item always gets the same ID, however many times it is scraped."""
return hashlib.sha256(f"{source}:{natural_key}".encode()).hexdigest()[:16]
def enqueue(db, urls):
"""Adding a URL that is already known is a no-op, so re-discovering links is harmless."""
db.executemany("INSERT OR IGNORE INTO frontier (url) VALUES (?)", [(u,) for u in urls])
def lease(db, lease_seconds=60):
"""Claim one URL. A lease left behind by a crashed worker expires and the URL is retried."""
now = time.time()
row = db.execute(
"UPDATE frontier SET status = 'leased', leased_until = ? WHERE url = ("
" SELECT url FROM frontier WHERE status = 'pending' OR (status = 'leased' AND leased_until < ?) LIMIT 1"
") RETURNING url", (now + lease_seconds, now)).fetchone()
return row[0] if row else None
def complete(db, url, new_urls=(), record=None):
"""Write the result and mark the URL done in one transaction: either both happen or neither does."""
db.execute("BEGIN IMMEDIATE")
try:
enqueue(db, new_urls)
if record:
now = time.time()
payload = json.dumps(record["data"], sort_keys=True)
digest = hashlib.sha256(payload.encode()).hexdigest()
db.execute(
"INSERT INTO records (record_id, url, content_hash, data, first_seen, last_changed) VALUES (?, ?, ?, ?, ?, ?) "
"ON CONFLICT(record_id) DO UPDATE SET data = excluded.data, content_hash = excluded.content_hash, "
"last_changed = excluded.last_changed WHERE records.content_hash != excluded.content_hash",
(record["id"], url, digest, payload, now, now))
db.execute("UPDATE frontier SET status = 'done' WHERE url = ?", (url,))
db.execute("COMMIT")
except Exception:
db.execute("ROLLBACK")
raise
def crawl(db, base, session=None, lease_seconds=60):
session = session or requests.Session()
enqueue(db, [f"{base}/list/0"])
while True:
url = lease(db, lease_seconds)
if url is None:
if db.execute("SELECT 1 FROM frontier WHERE status != 'done' LIMIT 1").fetchone():
time.sleep(1) # a lease is still held, perhaps by a worker that crashed; wait for it to expire
continue
return
html = session.get(url, timeout=30).text
if "/list/" in url:
links = [base + href for href in re.findall(r'href="(/(?:product|list)/\d+)"', html)]
complete(db, url, new_urls=links)
else:
sku = re.search(r'class="sku">([^<]+)<', html).group(1)
price = re.search(r'class="price">([^<]+)<', html).group(1)
complete(db, url, record={"id": record_id("example-shop", sku), "data": {"sku": sku, "price": price}})The crawl function is specific to our test site, with 25 listing pages linking to 500 product pages. Everything above it is reusable. Two details are easy to miss:
- The upsert only writes when content changed. The
WHEREclause on the conflict update leaves an unchanged record alone, solast_changedmeans what it says and can drive change detection. - The loop does not stop at an empty queue. Our first version ended when no URL could be leased, which left a page unfinished whenever a crash had left a lease outstanding. The loop now waits until every URL is done.
Killing it ten times
We ran a local test site with 25 listing pages and 500 product pages, so a complete crawl needs exactly 525 requests. Each run started the scraper, killed it with SIGKILL at a random moment between 0.2 and 1 second, and did that ten times before letting it finish. We ran the idempotent scraper above and the naive one described earlier, five times each, with the same kill timings, and a 3-second lease for the idempotent version.
| Run | Naive: rows stored | Naive: requests | Idempotent: rows stored | Idempotent: requests |
|---|---|---|---|---|
| 1 | 2,165 | 2,283 | 500 | 525 |
| 2 | 1,875 | 1,977 | 500 | 526 |
| 3 | 1,862 | 1,967 | 500 | 526 |
| 4 | 1,455 | 1,538 | 500 | 526 |
| 5 | 1,286 | 1,361 | 500 | 528 |
Both versions eventually stored all 500 products, because each finished its last run. The difference is everything else. The naive scraper stored between 2.6 and 4.3 rows per product and made between 2.6 and 4.3 times the necessary requests. The idempotent one stored exactly one row per product, every value matching the source page, and repeated at most three requests across ten crashes: one for each kill that landed while a page was in flight. It also finished sooner in every run, between 6.8 and 8.3 seconds against 9.2 to 10.8, even though it sometimes waited for a crashed worker’s lease to expire.
Beyond the database
Records are not the only thing a scraper does more than once. The same principle applies to every side effect:
- Files. Name downloaded images and documents by a hash of their content or of the record ID, so a repeat download overwrites instead of adding a copy.
- Messages and webhooks. Send an idempotency key derived from the record and its version, so a consumer can ignore a message it already processed.
- Counters and aggregates. Recompute them from stored records rather than incrementing on every fetch, or a restart inflates them.
- Raw responses. If you archive raw responses, a repeated fetch adds a second capture, which is harmless and even useful, as long as the extracted records stay idempotent.
Practical rules
- Pick the natural key carefully. It must be stable on the source: a SKU or listing ID, not a position on the page or a URL that carries tracking parameters.
- Make retries safe first, then frequent. Once writes are idempotent, retry and backoff can be generous without polluting data.
- Size leases to the slowest page. A lease shorter than a slow fetch lets two workers process the same URL; harmless here, but wasted bandwidth.
- Watch the repeat rate. Requests per stored record is a cheap health metric; a rise means crashes or lease problems, and belongs next to the others in monitoring a scraping pipeline.
The bottom line
A scraper that cannot be safely restarted will eventually corrupt its own data, and it will do so quietly. The fix is not more careful operation but a design that makes restarts boring: IDs derived from the data, a durable frontier, results and progress committed together, and leases that expire.
In our test, those four decisions turned ten crashes from up to 4.3 times the rows and requests into exactly the right 500 records and three repeated requests.