Labour-market intelligence looks like a scraping problem and is actually a measurement problem. The output is a time series that someone will read as a signal about hiring, and the failure mode is not a crashed job. It is a chart that moves for reasons that have nothing to do with the labour market.
If postings in one region drop eighteen percent this month, there are two explanations. Employers posted fewer roles, or your collection missed some. From inside the pipeline these are indistinguishable unless you built it to tell them apart. This is what separates building an index from pulling listings, and it is where most job-board panels quietly go wrong.
If your interest is recruiting workflows rather than aggregate measurement, the companion piece is residential proxies for recruiting and job market data. This one is about panels.
Coverage gaps look like trends
The defining property of a labour-market series is that it is compared against itself over time. Absolute counts matter far less than the delta, which means that any change in what you can see registers as a change in what exists.
Three things routinely shift coverage without shifting anything real:
Partial blocks. A board starts returning fewer results per query, or serves a challenge on the deeper pages of a result set. You still get data, so nothing looks broken, and your counts fall.
Geographic drift. The location your requests appear to originate from changes, and the result set changes with it. Postings are filtered by proximity on most boards, so this directly moves regional counts.
Silent pagination truncation. You collect three pages where you used to collect eight, because the later pages are defended more heavily than the first. Volume falls; nobody notices which pages disappeared.
Each of these produces a clean, plausible, wrong number. None of them raises an error.
Postings are served by location
Almost every major board resolves a query against where it thinks you are. Search “warehouse associate” without a location and you get regional results. Search with a location while arriving from an IP a thousand miles away and you get a result set that is neither what a local candidate sees nor a national view.
For a panel that is trying to measure regional hiring, this is the whole game. The location signal has to be stable across the entire history of the series, and it has to be consistent between the stated location parameter and the apparent origin of the request. Residential exits in the regions you are measuring make those two agree, which is what “a local view” means operationally.
The related trap is locale mismatch, where the IP says one country and the browser headers say another. That inconsistency changes result sets on international boards and is worth eliminating early. The details are in matching proxy geo, timezone and locale.
Design the panel, then collect it
The discipline that makes labour-market data usable is deciding what the panel is before writing a collector, and then not changing it casually.
Fixed source list. A defined set of boards, held constant. Adding a board mid-series creates a step change that will be read as hiring growth unless you backfill or version the series.
Fixed geography. A defined set of regions with a defined location parameter for each. Not “wherever the request came from”.
Fixed query set. The same occupational queries, run the same way, at the same cadence.
Fixed depth. A stated number of result pages per query, collected fully or marked as incomplete. Never best-effort.
Any change to any of these is a methodology change and should be recorded as one, with a version number stamped on every row. Analysts can work with a documented break in a series. They cannot work with an undocumented one.
Configuring the collection layer
With the Shifter gateway, region and session are encoded in the credentials against p.shifter.io:443:
customer-USERNAME-country-gb-city-manchester-sid-mcr01-ttl-600:PASSWORDcountry-gb uses the ISO alpha-2 code, which is gb rather than uk. city-manchester narrows the exit to the metro whose postings you are measuring. sid-mcr01 holds a sticky session so that a full query, including its pagination, runs from a single IP, and ttl-600 keeps that IP for ten minutes. A session identifier per region and query, rather than per request, is what keeps a paginated result set internally coherent.
If a region and city filter is too narrow to fill, the gateway returns 502 rather than substituting a nearby exit. For a measurement panel that is the correct behavior: a missing observation you can see beats a silent substitution you cannot.
Cadence should be steady and unremarkable. Daily or weekly collection at consistent times, with modest concurrency and real backoff on errors, produces a cleaner series than aggressive sweeps, and it is far less likely to change your coverage by provoking defenses. The mechanics are in rate limiting and request throttling.
Deduplication is a measurement decision
The same role commonly appears on several boards, on a company careers page, and again two weeks later as a repost. How you resolve that determines what your index actually measures, and there is no neutral choice.
A workable approach is a composite key of normalized employer, normalized title, location and posting date window, with a decision recorded for each collapse. What matters more than the specific algorithm is that the rule is fixed, documented and applied identically across the whole history. Changing the dedupe logic retroactively rewrites the past, and a labour-market series whose past keeps changing is not a series.
Reposts deserve their own treatment. A role reposted monthly is either a hard-to-fill vacancy or an evergreen listing, and both are interesting, but only if you can tell them from new demand.
Verify coverage instead of assuming it
The single practice that separates panels you can defend from panels you cannot: measure your own collection, not just the postings.
Run a small control set of queries twice per collection window, from two different exits in the same region, and compare result counts. Convergence means your view is stable. Divergence means the boards are responding to something about your requests, and the series for that window is suspect.
Track the operational metrics alongside the data itself: success rate per board, pages collected against pages expected, and result counts per query over time. A twenty percent drop in postings that coincides with a drop in your own success rate is a collection story, not a labour-market story, and you want to know which one you are looking at before someone builds a forecast on it.
For teams that need a defensible baseline on the collection layer itself, testing proxy speed, success rate and location accuracy covers the measurement side.
Staying on the right side of the data
Job postings are published for people to read, which is a genuine argument for collecting them, and it is not unlimited.
Postings often contain named recruiters, direct phone numbers and email addresses. That is personal data, and an index of hiring demand does not need it. Strip it at ingestion rather than storing it and promising not to use it. Aggregate what you publish, respect each board’s stated terms, keep request volumes proportionate, and identify your traffic honestly where a board provides a route for research access.
The general framing is in ethical residential proxies for AI data collection, and it applies here with more force, because labour-market work sits closer to individuals than most commercial data collection does.
FAQ
How many regions does a credible national panel need?
Enough that no single metro dominates the aggregate, and consistent from the first observation onward. Ten well-chosen regions collected identically for a year beats forty collected unevenly for three months.
Should I collect salary fields?
Yes, and store the raw string alongside your parsed value. Salary disclosure varies by jurisdiction and by board, so a rising share of postings with salary data is often a policy change rather than a market one, and you need the raw text to tell.
Why not use a single board with a public API?
Because one board’s coverage is one board’s business, and its share of postings shifts over time. A single-source series measures that source. Whether that is acceptable depends on what you are claiming.
Does this work for international coverage?
It does, with per-country exits and locale settings that match. Expect national boards to dominate in most markets, which changes the source list rather than the method. Related context is in the recruitment use case.
The bottom line
Job-board data becomes labour-market intelligence at the point where you can explain every movement in the series, including the ones your own pipeline caused. That requires a fixed panel, geographically consistent collection, a stable dedupe rule and honest coverage metrics.
Residential proxies are the piece that makes the geography real and repeatable, so that a regional number means what it says. The rest is methodology, and methodology is what makes the number worth quoting. Bandwidth planning for a panel of this shape is covered in estimating monthly residential proxy bandwidth, with rates on the residential proxy pricing page.