Investment firms use web data to see what is happening before it shows up in a quarterly report: prices changing across thousands of products, job postings rising or falling, store openings, reviews, app rankings, shipping times. Collected well, this alternative data is a real edge for financial research teams. Collected badly, it is a regulatory and legal risk that can follow a fund for years.
Most writing about alternative data covers what to collect. Much less covers the compliance layer around it: what a fund’s compliance team needs to know before data can be used in an investment decision, and what a data provider needs to be able to prove. This guide covers both. It is general information, not legal advice; investment firms should involve their own compliance and legal counsel.
Why alternative data needs its own compliance review
Ordinary market data comes from exchanges and licensed vendors with long-established rules. Web data comes from sources that never agreed to be an input to anyone’s trading. That raises questions compliance teams have to answer before the data is used:
- Could it contain material non-public information? Data is only safe to trade on if it was obtained lawfully from public sources and does not breach a duty of confidentiality. Investment advisers in the US must keep written policies designed to prevent the misuse of material non-public information, and data sources fall within those policies.
- Was it collected lawfully? Terms of service, logins, technical restrictions and data protection law all apply to how data was gathered, not only to what it is.
- Does it contain personal data? Many web datasets touch individuals: reviewers, employees, app users. European data protection law applies to that data whether or not it was public.
- Is it what the vendor says it is? A dataset’s value depends on how it was built. Misdescribing that is not just a commercial problem.
That last point became a regulatory one in 2021. The US Securities and Exchange Commission charged App Annie, a provider of mobile app data, and its co-founder with securities fraud for misrepresenting how its data was derived. The SEC said the company had used non-aggregated, non-anonymised data to alter the estimates it sold to trading firms, despite telling customers otherwise. App Annie agreed to pay a 10 million dollar penalty. It was the SEC’s first enforcement action charging an alternative data provider with securities fraud, and it put the methods behind a dataset, not only its content, under scrutiny.
What a fund’s compliance team will ask
Buy-side firms have converged on a fairly standard set of questions for data vendors. The financial data industry body FISD publishes a due diligence questionnaire through its Alternative Data Council, used by many funds as a starting point, with versions for full vendor reviews, short data trials and, more recently, generative AI. The questions fall into a few groups:
| Area | What they want to know |
|---|---|
| Source | Where the data comes from, and whether each source permits this use |
| Collection | How it is collected: public pages or behind logins, which methods, at what rate |
| Rights | Whether the vendor has the right to license the data, and what the licence allows |
| Personal data | Whether any is included, how it is filtered or anonymised, and the legal basis |
| MNPI | Controls that keep confidential or non-public information out of the dataset |
| Methodology | How raw data becomes the delivered product, and how changes are communicated |
| Governance | Policies, audits, complaint handling and what happens if a source objects |
If you build alternative data for funds, expect to answer every row in writing. If you work at a fund collecting data yourself, your compliance team will ask the same questions of you.
The four controls that matter most
1. Public sources, collected without breaching access controls. Collect what any visitor can see without logging in, accepting special terms or getting around technical restrictions. Data obtained by misusing an account or getting past a paywall is exactly the kind compliance teams cannot accept. Our guide to what your legal team will ask before approving a scraping project covers the collection questions in detail.
2. Personal data designed out. Most investment signals do not need individuals. Count job postings without storing applicants; aggregate reviews without keeping reviewer identities; measure app rankings without user data. Filtering at collection time is far easier to defend than anonymising later. Our GDPR overview covers the basics for collection teams.
3. A documented, honest methodology. Write down how the raw data becomes the product: sources, sampling, cleaning, weighting, imputation and any modelled estimates. When the method changes, say so to customers. The App Annie case shows that the gap between the methodology described and the one used can itself be the violation.
4. Provenance you can prove. Keep, for every record, the source, the collection time and the method, and keep raw captures for a reasonable period. When a fund asks where a number came from, or a regulator does, you need to show it rather than describe it.
Rights: what you can license
A data provider can only license what it has the right to license. Facts such as prices and counts are generally not owned by anyone; creative content such as article text and photos belongs to its authors; and in Europe, some databases are protected even when they contain only facts. Selling derived signals, such as indices, counts and trends, is very different from redistributing copies of a website’s content. Our guide to who owns scraped data covers these layers and what they mean for resale.
The infrastructure question
Funds increasingly ask about the infrastructure behind collection too, including proxy networks. The concern is reasonable: if a provider’s IP addresses come from devices whose owners did not agree to share their connection, the dataset inherits that problem. Ask your proxy provider how its addresses are sourced, what consent the device owners gave, and how abuse is handled. Our explainer on how providers ethically source residential IPs lists what to ask. For collection itself, residential proxies are often needed to see prices and listings as local customers see them, which matters for any signal built on what consumers actually pay.
Putting it together
A practical compliance pack for an alternative dataset fits in a few documents:
- A source register: every source, its access type, its terms and its robots.txt position, checked on a stated date.
- A collection description: methods, rates, and a statement that no logins, paywalls or access controls are bypassed.
- A personal data assessment: what was excluded, how, and the legal basis for anything kept.
- A methodology document, with a change log.
- A provenance record: per-record source and time, with raw captures retained.
- A complaint and takedown procedure: who handles an objection from a source, and how quickly collection from it can stop.
With that pack, a vendor can answer a due diligence questionnaire in days, and a fund can show its regulators how a signal was built. Our earlier article on residential proxies for alternative data covers the collection side of the same work.
FAQ
Is using scraped web data for investing legal?
Generally, yes, when the data is public, collected lawfully and free of material non-public information and unlawfully processed personal data. The details depend on the jurisdiction, the source and how the data was collected, so funds review each dataset before using it.
What is a due diligence questionnaire for alternative data?
A standard set of questions that investment firms send to data vendors about sources, collection methods, rights, personal data, MNPI controls, methodology and governance. The FISD Alternative Data Council publishes a widely used version.
Why did the SEC charge an alternative data provider?
In 2021 the SEC charged App Annie and its co-founder with securities fraud for misrepresenting how its data was derived, and the company agreed to pay a 10 million dollar penalty. It was the SEC’s first such case against an alternative data provider.
Can alternative data contain personal data?
It often can, through reviews, job postings or profiles. The safest approach is to design personal data out at collection time and keep only aggregated signals.
The bottom line
Alternative data is only an edge if it survives compliance review. Collect from public sources without getting around access controls, design personal data out, document the methodology honestly and keep provenance for every record. Vendors who can answer a fund’s due diligence questions in writing win the business; funds that ask those questions avoid inheriting someone else’s problem.
Sources and references
- U.S. Securities and Exchange Commission, SEC Charges App Annie and its Founder with Securities Fraud, 14 September 2021.
- FISD, Alternative Data Council, including its due diligence questionnaires.