Brands want to know what people say about them, their products and their competitors, and much of that conversation happens in public: forums, review sites, community boards and public social profiles. Collecting it is the basis of social listening, and it is harder than it looks. Every platform structures its pages differently, many render content with JavaScript, and almost everything you collect involves people.
This guide covers how teams collect public social data today with web scraping APIs: which sources to use and when official APIs are the better route, how to turn very different platforms into one dataset, and how to keep personal data out of it by design.
Start with what you are trying to learn
Social data projects go wrong when they collect everything and decide later. Decide first which question you are answering:
- Volume: how often is the brand mentioned, and is that changing?
- Topics: what are people saying, about which products or issues?
- Comparison: how do mentions of the brand compare with competitors’?
- Incidents: is a complaint or a counterfeit listing spreading?
Each question needs only a few fields: the post’s link, time, text and source. Few projects need anything more about the people who posted.
Choose the source route
| Source | Best route |
|---|---|
| Platforms with an official API that covers your need | The official API, within its terms |
| Public forums, community boards and review sites | Collection of public pages |
| Pages that render with JavaScript | A web scraping API with rendering |
| Anything behind a login | Do not collect it |
Official APIs are usually the most reliable and clearly permitted route where they exist and fit your volume and budget. For public pages without one, a web scraping API handles the parts that make social sites awkward: rendering JavaScript, waiting for content to load, and fetching pages as they appear in a given country. Extraction rules return the fields you need as JSON instead of raw HTML.
Stay on the public side of every site. Content behind a login, private groups and direct messages are off limits, both because accessing them usually means accepting terms that forbid collection and because the people involved expect privacy.
One schema for every platform
Each source names its fields differently: a forum has a permalink and a body, a review site has a link and review text, and dates come in different formats and time zones. If you analyse them separately, comparisons break. Map every source to one small schema as you collect:
- source: where the post came from;
- url: the canonical address, so the same post counts once;
- published_at: the time in UTC;
- text: the post’s text, whitespace normalised;
- author_key: a pseudonymous key instead of the author’s name.
The module below does that mapping, removes duplicate URLs (tracking parameters and fragments often make one post look like several) and counts daily mentions per brand:
import hashlib
from collections import Counter
from datetime import datetime, timezone
from urllib.parse import urlsplit, urlunsplit
# Where each source keeps the fields we need, after extraction rules have run.
FIELD_MAPS = {
"forum": {"url": "permalink", "text": "body", "author": "username", "time": "created"},
"review": {"url": "link", "text": "review_text", "author": "reviewer", "time": "date"},
}
def canonical(url):
"""Drop query strings and fragments so the same post counts once."""
parts = urlsplit(url)
return urlunsplit((parts.scheme, parts.netloc.lower(), parts.path.rstrip("/"), "", ""))
def normalise(source, raw, salt):
"""One post in a common shape, with the author pseudonymised and no other personal data."""
m = FIELD_MAPS[source]
when = datetime.fromisoformat(raw[m["time"]].replace("Z", "+00:00")).astimezone(timezone.utc)
return {
"source": source,
"url": canonical(raw[m["url"]]),
"published_at": when.isoformat(timespec="seconds"),
"text": " ".join(raw[m["text"]].split()),
"author_key": hashlib.sha256((salt + raw[m["author"]]).encode()).hexdigest()[:16],
}
def daily_mentions(posts, brands):
"""Count posts per day that mention each brand, counting each URL once."""
seen, counts = set(), Counter()
for post in posts:
if post["url"] in seen:
continue
seen.add(post["url"])
day = post["published_at"][:10]
for brand in brands:
if brand.lower() in post["text"].lower():
counts[(day, brand)] += 1
return dict(sorted(counts.items()))Applied to three illustrative records, one of them a duplicate with a tracking parameter:
import os
from social import daily_mentions, normalise
salt = os.environ.get("MENTIONS_SALT", "rotate-me-per-project")
raw = [
("forum", {"permalink": "https://forum.example/t/88?ref=home", "body": "Acme kettle died after a week",
"username": "kitchen_fan", "created": "2026-10-20T08:15:00Z"}),
("forum", {"permalink": "https://forum.example/t/88", "body": "Acme kettle died after a week",
"username": "kitchen_fan", "created": "2026-10-20T08:15:00Z"}),
("review", {"link": "https://reviews.example/p/42", "review_text": "Better than my old Brandly one",
"reviewer": "J. D.", "date": "2026-10-20T19:40:00+02:00"}),
]
posts = [normalise(source, item, salt) for source, item in raw]
print(daily_mentions(posts, ["Acme", "Brandly"])){('2026-10-20', 'Acme'): 1, ('2026-10-20', 'Brandly'): 1}The duplicate counts once, the review’s local time is converted to UTC, and no usernames reach the output.
Personal data, designed out
Public posts are still personal data in Europe and many other places, and data protection rules apply however public the post was. The safest design keeps as little about people as the question needs:
- Pseudonymise authors. A salted hash lets you count distinct posters or spot one account posting repeatedly, without storing the username. Use a different salt per project, and keep it secret.
- Do not collect profiles. Follower counts, bios, photos and locations of individuals are rarely needed for brand questions.
- Keep retention short. Aggregate counts and topics can be kept; raw text can usually be deleted after analysis.
- Report in aggregate. Publish trends and examples without identifying people.
Our guide to scraping social platforms for sentiment analysis covers the analysis side once the data is clean, and our guide to tracking brand mentions online covers building the query set and handling regional differences.
Collect gently
Social and community sites are run for their users. Collect at a pace a reader might browse, prefer the pages that list new posts over crawling whole archives, and stop collecting from any source that asks you to. Brand monitoring rarely needs more than a daily pass over the sources that matter.
FAQ
Is it legal to collect social media data?
Collecting public posts is often lawful, but platform terms, how you access the data and data protection law all apply. Use official APIs where they fit, stay away from content behind logins, and keep personal data to what the purpose needs. Take legal advice for your specific case.
Should I use a platform’s API or a web scraping API?
Use the official API when it covers the data and volume you need within its terms. Use a web scraping API for public pages that have no suitable API, especially pages that render with JavaScript.
How do I avoid collecting personal data from social media?
Collect only the fields your question needs, replace usernames with salted pseudonymous keys, skip profiles entirely, keep raw text only as long as needed, and report results in aggregate.
How often should I collect social mentions?
Daily is enough for most brand monitoring. Collect more often only during a launch or an incident, and focus on the sources where your audience actually posts.
The bottom line
Social data is valuable because it is what real people say in public, and that is also why it needs care. Decide the question first, use official APIs where they fit and a web scraping API for public pages, map every source into one small schema, and design personal data out from the start. The result is a dataset that answers brand questions without collecting more than it should.