Most B2B lead databases fail in one of two ways, and neither is a scraping failure. They decay, because people change jobs and companies change shape faster than anyone refreshes the records. Or they become a liability, because nobody can say where a contact came from, on what basis it is held, or whether that person asked to be removed.
Both failures are design decisions made on day one. This guide covers building a lead database from public sources so that it stays accurate and stays defensible. The case for why proxy infrastructure matters to lead generation in the first place is in residential proxies for B2B lead generation. This is the build.
Start from the ideal customer profile, not the scraper
The ideal customer profile decides which sources matter, which fields you need and how large the database should be. Write it down before any collection starts: industries, size bands, regions, business models, and the observable signals that indicate fit.
A clear profile keeps the database small and relevant. A vague one produces a large database of companies nobody will ever sell to, all of which still carry the same compliance obligations as the ones you do.
Build the account universe first
Companies are the stable layer. Build them before any contacts, from sources that describe organisations rather than people.
| Source | What it gives you | Notes |
|---|---|---|
| Business registries | Legal name, registration number, status, registered address | Authoritative; many offer bulk files or APIs |
| Industry and association directories | Membership lists by sector and region | Often paginated and rate-limited |
| Conference exhibitor and sponsor lists | Companies investing in your category | Time-bound and highly relevant |
| Marketplace and partner directories | Companies integrating with a given platform | Strong technographic signal |
| Company websites | Products, locations, pricing, customers | The richest source, and the most varied |
| Job postings | Growth, team structure, tools in use | A leading signal; see job-board data |
Where a registry or directory publishes bulk data or an API, use it. Scrape what is not offered any other way.
Resolve company identity before anything else
The same company appears under a legal name, a trading name, several domains, and as a parent and subsidiaries. If identity is not resolved, the database fills with duplicates, and every downstream count is inflated.
Use the primary web domain as the working key, since it is the identifier most sources share. Normalise legal names by stripping suffixes and punctuation, attach registry identifiers where you have them, and model parent and subsidiary relationships explicitly rather than merging them. Measure the duplicate rate on a hand-checked sample and keep it visible.
Firmographics and signals, each with a source
For every company field, store three things: the value, where it came from, and when it was observed. Industry, size band, locations, technology indicators and hiring activity all change, and a value without a date cannot be trusted six months later.
Signals such as hiring spikes or new product launches are events rather than attributes, and they lose value quickly. The case for treating them that way is in scraping intent signals to power sales intelligence.
The contact layer: roles first, people second
Contacts are where both decay and compliance risk concentrate, so add them last and add them carefully.
Model roles before people. “Head of data engineering at this company” stays true long after the person holding it moves on. Store the role as the durable record and attach the current person to it as an observation with a date. The same principle is set out in building talent-mapping and org-chart data.
Keep to business context. Name, role, company and a work email or business phone published in a professional context. Personal email addresses, personal phone numbers, home addresses and social profiles unrelated to work do not belong in a B2B database.
Collect only what is public, and only where you are allowed. Do not scrape content behind a login, and respect each source’s terms.
Do not probe mail servers to guess addresses. Generating likely addresses and testing them against a company’s mail server is widely treated as abusive, damages your sending reputation, and adds nothing you can defend. If you need verified addresses, use a verification service you have vetted, or rely on addresses people published themselves.
Design compliance into the schema
Compliance that lives in a policy document and not in the data model does not survive contact with a real database. Put it in the tables.
| Field | Purpose |
|---|---|
| Source URL and source type | Proves where each value came from |
| Collected at | Supports freshness and retention decisions |
| Lawful basis reference | Links the record to the documented basis for holding it |
| Purpose | Limits use to what the basis covers |
| Jurisdiction, where known | Drives which rules apply to outreach |
| Notice status | Records whether and when the person was informed |
| Retention expiry | Forces review or deletion |
| Suppression flag | Stops all further processing and outreach |
A few points of law shape those fields. They are general, and you should work with your own counsel.
- Business contact details are personal data. Under GDPR, a named person’s work email is still personal data, and the regulation applies.
- Legitimate interest has to be documented. It is the usual basis for B2B prospecting, and it requires a written balancing assessment, not an assumption.
- People must be told. When you obtain someone’s data from sources other than them, GDPR generally requires informing them, at the latest at first contact when you use it to communicate.
- California no longer exempts B2B data. Since 2023, business contact information has been in scope of California’s privacy law.
- Outreach rules vary. Rules for unsolicited email to business addresses differ by country, and some require prior consent even for B2B. US commercial email rules require a working opt-out.
The suppression list is permanent and global. When someone opts out or asks to be deleted, remove them from every derived table, cache and export, and keep a minimal record that prevents them from being re-collected. A suppression that is undone by the next crawl is not a suppression.
The wider framing is in residential proxies and GDPR compliance.
Collecting at scale
Different sources move at different speeds, so give each its own cadence rather than one global crawl.
| Source | Typical cadence |
|---|---|
| Registries | Monthly, or on their own publication schedule |
| Directories and association lists | Weekly to monthly |
| Company websites | Monthly, with change detection |
| Job postings | Daily |
| Event lists | When published, then frozen |
Directories are usually paginated and throttled. Hold one exit across a full walk so pagination stays consistent, and rotate for independent page fetches. With the Shifter gateway, both are set in the credentials against p.shifter.io:443:
customer-USERNAME-country-de-sid-dir-assoc-07-ttl-600:PASSWORDRegional directories and registries often serve differently by location, so collect each region from that region. Keep request rates ordinary and back off on errors, as in rate limiting and request throttling. For directories that load results with JavaScript, a rendered request is often simpler than running your own browsers; see when you need a web scraping API.
Freshness is a process
A lead database decays continuously. Build the refresh into operations rather than scheduling an annual clean-up.
- Re-verify contacts on a schedule, and mark any person-to-role link older than your threshold as stale.
- Re-crawl company sources on their cadence, and record what changed.
- Feed outreach results back. Bounces, opt-outs and “no longer here” replies are the most accurate freshness signal you will get, and they must update the database, not just the outreach tool.
Measuring the database
| Metric | What it tells you |
|---|---|
| Coverage of the ideal customer profile | Whether the database contains the market you sell to |
| Duplicate rate | Whether identity resolution is working |
| Field completeness | Where sources are thin |
| Staleness distribution | How much of the data is past its refresh threshold |
| Bounce and opt-out rates | Real-world accuracy and consent health |
| Suppression hits during collection | Whether suppressed people are being re-collected |
FAQ
Is scraping B2B contact data legal?
It can be, when the data is public, business-context only, held on a documented lawful basis, and handled with transparency and opt-outs. The answer depends on jurisdiction and use, so involve counsel.
Can we scrape professional networking platforms?
Not behind a login, and not against their terms. Build from company sites, registries and directories instead.
Should we buy data rather than build it?
Licensed data complements a build well. The compliance obligations for how you use it still sit with you.
How often should contacts be re-verified?
Often enough that the stale share stays small. Many teams re-check active target accounts quarterly and let outreach feedback flag the rest.
The bottom line
A B2B lead database that lasts is an account model first and a contact list second. Build companies from authoritative sources, resolve their identity, store every value with its source and date, attach contacts at the role level with business-context data only, and put lawful basis, notice, retention and suppression in the schema rather than in a policy.
Collect each source at its own cadence, keep refresh running continuously, and feed outreach results back into the data. The product view is on the lead generation data collection page, and the enrichment side is covered in contact and company enrichment.