Most plans for collecting web data budget for scrapers and forget the people. The first version takes a few weeks to write, so a team of one looks fine. Then sites change, volume grows, a customer finds a wrong number, and it turns out that the job was never writing scrapers. It was keeping them true.
We could not find a credible published benchmark for how many people this takes, and any single ratio would mislead you anyway. So this guide offers the roles, a small model you can run with your own numbers, and what tends to break first at each size.
What drives headcount
Data volume matters less than you would think. Fetching ten times more pages from the same sources is mostly a budget and infrastructure question. Headcount follows four things:
- The number of sources, since each one can break on its own.
- How often they break, which depends on how often a site changes and how hard it is to access.
- How wrong you can afford to be. A weekly research dataset can wait. A price feed that customers act on cannot.
- How many new sources you add, because onboarding competes with maintenance for the same people.
The roles
These are roles, not job titles. On a small team one person holds several, which is fine as long as each has a name next to it.
| Role | Owns | Often missing until |
|---|---|---|
| Collection engineer | Scrapers, extraction, fixing breakage | Never missing |
| Platform engineer | Scheduling, queues, proxies, browsers, storage, cost | Several people are repeating the same plumbing |
| Data quality owner | Checks, drift, what “correct” means | A customer finds the error first |
| Product or data owner | What to collect, priorities, source decisions | Everything is urgent |
| Legal and compliance contact | Scope, terms, personal data, approvals | A new source raises a question nobody can answer |
The last one is rarely full-time, but it should be a named person. A short written data collection policy saves that person from answering the same questions repeatedly.
A planning model
The script turns recurring work into hours and hours into people. It needs nothing beyond Python 3.9 or later. Every number is an assumption to replace with your own: count how many fixes you did last quarter, and how long they took.
"""A planning model for how many engineers a data collection operation needs.
Every default is an assumption to replace with your own numbers.
"""
from __future__ import annotations
import math
HOURS_PER_FTE = 30 # focused engineering hours per person per week
def team_size(sources, breaks_per_source_per_month, hours_per_fix, new_sources_per_month,
hours_per_new_source, weekly_platform_hours=20, weekly_support_hours=8):
"""Recurring hours become people; reserve a quarter of the time for project work."""
fixes = sources * breaks_per_source_per_month * hours_per_fix / 4.33
onboarding = new_sources_per_month * hours_per_new_source / 4.33
weekly = fixes + onboarding + weekly_platform_hours + weekly_support_hours
fte = weekly / (HOURS_PER_FTE * 0.75)
return {"fixes": round(fixes), "onboarding": round(onboarding),
"platform": weekly_platform_hours, "support": weekly_support_hours,
"weekly_hours": round(weekly), "fte": round(fte, 1), "people": math.ceil(fte)}
if __name__ == "__main__":
scenarios = {
"10 sources, stable": (10, 0.25, 4, 1, 16),
"50 sources, mixed": (50, 0.5, 4, 3, 16),
"200 sources, mixed": (200, 0.5, 4, 8, 16),
"200 sources, hard": (200, 1.5, 6, 8, 24),
}
print(f"{'scenario':22}{'fixes':>6}{'new':>5}{'platform':>9}{'support':>8}{'hrs/wk':>7}{'FTE':>6}{'people':>7}")
for name, args in scenarios.items():
r = team_size(*args)
print(f"{name:22}{r['fixes']:>6}{r['onboarding']:>5}{r['platform']:>9}{r['support']:>8}"
f"{r['weekly_hours']:>7}{r['fte']:>6}{r['people']:>7}")Output:
scenario fixes new platform support hrs/wk FTE people
10 sources, stable 2 4 20 8 34 1.5 2
50 sources, mixed 23 11 20 8 62 2.8 3
200 sources, mixed 92 30 20 8 150 6.7 7
200 sources, hard 416 44 20 8 488 21.7 22The scenarios use illustrative inputs rather than measurements, and the arguments in the script show each one. Stable sources break once every four months and take four hours to fix; hard ones break one and a half times a month and take six.
Two things stand out. Going from 50 to 200 mixed sources multiplies the source count by 4, but team size by about 2.4, because platform work is shared. Then the same 200 sources go from 7 people to 22 purely because they are harder to keep running. Breakage rate moves headcount more than source count does, so that is the number to measure first.
The model keeps a quarter of each person’s time for project work. That is lean. Google’s SRE book sets a goal of at least 50% of time on engineering rather than operational work, and at that rate these teams would be 1.5 times larger. If nobody has time to fix root causes, the same breakages keep returning.
What breaks first
One to two people. The bus factor. Everything lives in one head, and holidays become incidents. Write down how each source works and how to fix it. Buy your infrastructure rather than running it.
Three to five people. Quality. The team is busy fixing visible failures while silent ones build up, like a selector that returns the wrong field with a 200. Name a quality owner and monitor the pipeline on content as well as on status codes.
Six to ten people. Duplication and priorities. Each engineer has built their own retries and proxy handling, and every stakeholder’s source is urgent. This is when a platform owner pays off, along with a product owner who can say no.
More than ten. Coordination and on-call. Google’s SRE book calculates that a single-site team with two people on-call and week-long shifts needs at least eight engineers. Few data teams should build that. Design pipelines to recover alone and catch up afterwards, and keep round-the-clock cover for what customers truly depend on.
Reducing the number you need
- Buy the commodity layers. Our build versus buy guide covers what running your own infrastructure really takes.
- Cut sources. Retiring the ten least valuable sources is often the cheapest hire you can make.
- Prefer stable routes. Official APIs, feeds and structured data break less than HTML selectors.
- Fix classes of failure. Shared retries, shared checks and shared parsers turn many small fires into one.
The bottom line
Do not staff from a ratio. Staff from your source count, your breakage rate and how wrong you can afford to be, and measure those numbers before the next hiring discussion. Name an owner for quality early, buy what is not your advantage, and expect the bottleneck to move as you grow.
Sources and references
- Google, Site Reliability Engineering, Being On-Call: sre.google
- Google, Site Reliability Engineering, Eliminating Toil: sre.google