Knowledge

Scraping Intent Signals to Power Sales Intelligence

Buyer intent data is mostly public if you know which signals to read. What the useful ones are, how fast they decay, and how to collect them at scale.

James Meadow

James Meadow

September 6, 2026 · 8 min read

Most sales teams first meet buyer intent data as something you buy. A vendor sells a score, the score says an account is in market, and nobody in the building can explain what produced the number.

That is one kind of intent data. There is another kind sitting in public view, and it has a property the purchased kind usually lacks: you can point at the specific event that caused it. A company posted a role. A company changed its pricing page. A company appeared in a review-site category it was not in last quarter. Those are observable facts with timestamps, and a sales team can act on a fact in a way it cannot act on a score of 78.

What buyer intent data actually is

The term covers three different things that get discussed as though they were one.

First-party intent is behavior on your own properties. Someone read three pricing pages, downloaded a comparison, came back twice. You own this data, it is the most reliable kind, and it only exists for accounts that already found you.

Third-party intent is behavior observed across a publisher network or co-op, sold as an aggregate. It has reach that first-party data does not, and the trade-off is that you generally cannot see the underlying events or verify the methodology.

Public-web intent is the observable footprint a company leaves as it changes. Hiring, technology, funding, product, pricing, public complaints. Nobody sells it as a score because it is not a score. It is a stream of events you have to collect and interpret yourself.

The third category is the one this article is about, because it is the one where a collection layer determines what you can see.

The signals worth collecting

Not everything public is a signal. The ones that reliably precede a buying conversation share a property: they cost the company something to produce, so they are hard to fake and they indicate real commitment.

Job postings. The strongest single public signal. A company hiring three data engineers is building something. A company hiring its first compliance officer has crossed a regulatory threshold. Postings also name tools directly, which turns a vague signal into a specific one. The collection mechanics for this surface are in job-board data and labour-market intelligence.

Technology changes. New tags, new subdomains, a migration visible in public DNS or page source. Adding a category of tooling usually means budget already moved.

Pricing and packaging changes. A new tier, a removed plan, a shift from per-seat to usage-based. These often precede a wave of comparison shopping by that company’s own customers, which is an intent signal about a whole market segment rather than one account.

Review-site movement. A company appearing in a new category, or accumulating complaints in an existing one, is a displacement signal. Public dissatisfaction with an incumbent is the most direct buying intent that exists.

Funding and leadership changes. Well covered by existing data vendors, so the marginal value of collecting it yourself is low. Worth having, not worth building for.

Public questions. Community posts asking how to solve a problem your product solves. High intent, low volume, and the hardest to do well because the signal-to-noise is poor.

Recency is most of the value

The mistake that wastes the most effort is treating intent signals as attributes rather than events.

A job posting from last week is a reason to call. The same posting from six months ago is background. If your pipeline stores signals without decay, your scoring drifts toward companies that generated a lot of signal at some point rather than companies doing something right now, which is close to the opposite of what intent data is for.

The practical consequence is that cadence determines value. A signal that changes daily, collected weekly, is a lagging indicator no matter how good the collection is. Decide the refresh rate per signal type against how fast that signal actually moves, and be honest that a slow crawl of a fast signal is not intent data, it is history.

Two implications follow. Store the observation timestamp, not just the signal. And apply an explicit decay function when scoring, so an account’s score falls on its own when nothing new happens.

Why collection is the hard part

These signals live on surfaces that were not built to be read in bulk, and three properties make them awkward.

They are geo-filtered. Job boards return results by proximity. Review sites and app stores localize both catalogue and content. A company’s German postings may be invisible from a US vantage point, which quietly turns an international coverage claim into a domestic one.

They are paginated and defended. The interesting results are rarely on page one, and deeper pages are usually where rate limiting starts.

They are the same page for everyone until they are not. Personalization and A/B tests mean two collectors can disagree about what a page said, which matters when you are recording an event.

This is where a residential proxy layer stops being an implementation detail. Geo-consistent exits let a regional signal actually be regional. Sticky sessions keep a paginated result set internally coherent, so page four belongs to the same query as page one.

With the Shifter gateway, both go in the credentials against p.shifter.io:443:

customer-USERNAME-country-de-city-berlin-sid-intent01-ttl-600:PASSWORD

country-de and city-berlin set the vantage point, sid-intent01 holds one exit across the whole query, and ttl-600 keeps it for ten minutes. One session per query, not per request, is the rule that keeps a result set coherent.

Cadence should look like interest rather than a sweep: spread collection across the day, keep concurrency modest, and back off on errors rather than retrying through them. The mechanics are in rate limiting and request throttling.

Turning signals into something a rep can use

A stream of events is not sales intelligence. Three steps close the gap.

Resolve to accounts. Normalize company names, domains and subsidiaries so that a posting, a review and a tech change land on one record. This is unglamorous and it is where most in-house intent projects stall.

Weight by specificity, not volume. One posting naming a competitor’s product is worth more than forty generic postings. Scoring that counts events rewards large companies for being large.

Deliver the evidence, not the score. A rep can open a conversation with “I saw you are hiring two people to run this” and cannot do anything with “your account scored 78.” Pass the underlying event through to the CRM record. This single choice does more for adoption than any modelling improvement.

The broader account-data pipeline this feeds is covered in residential proxies for B2B lead generation, and the product view is on the lead generation page.

Where the line is

Public-web intent collection is about companies, and it stays clean as long as it stays there.

Collect what organizations publish about themselves. Do not collect personal information about individuals to build behavioral profiles of them, do not scrape content behind a login or an account you agreed not to automate, and do not treat a named recruiter’s phone number in a job posting as an asset because it was technically visible. Strip personal identifiers at ingestion rather than storing them and promising restraint later.

The distinction is not subtle: a company hiring data engineers is a business fact, and a named individual’s activity is a person. The first is intent data. The second is surveillance with a sales deck attached. The general framing is in ethical residential proxies for AI data collection, and it applies with equal force here.

FAQ

Is public-web intent better than purchased intent data?

It is more explainable and more verifiable, and it has less reach. Most teams that do this well use both, with purchased data for breadth and public signals for the accounts they are actually working.

How often should signals be refreshed?

Match the refresh rate to how fast the signal moves. Job postings and review activity reward daily or near-daily collection. Technology and pricing changes are fine weekly. Funding is event-driven and better bought.

What is the minimum viable version?

One signal, collected properly, resolved to accounts, delivered into the CRM with the evidence attached. Job postings are the usual starting point because the signal is strong and the surface is well understood.

Do I need geo-targeting if we only sell domestically?

If your accounts are domestic, one vantage point can be enough. The moment you claim international coverage, an unpinned collector will under-report foreign markets in a way that looks like those markets being quiet.

The bottom line

Buyer intent data is not a product you have to buy. A large part of it is public, event-shaped and dated, which makes it explainable in a way a purchased score never is.

The work is in the collection layer and the plumbing: seeing regional signals from the right vantage point, refreshing fast enough that a signal is still actionable when it lands, resolving events to accounts, and passing the evidence through instead of a number. Bandwidth planning for that pipeline is on the pricing page.

Ready to get started?

Try Shifter's residential proxies, 205M+ IPs, 195+ countries, from $0.10/GB.

Get Started