“Real-time housing data” sounds like a latency problem, and it is not. Housing has no price tick. Transactions are sparse, individually negotiated, and recorded in public registers weeks or months after they complete. Nothing about the underlying market moves in seconds.
What does move quickly is the listing surface: a property appears, its price changes, it goes under offer, it is withdrawn, it comes back. Those events happen daily and they lead the transaction record by a wide margin. A real-time housing feed is therefore an event-detection system, and its job is to notice changes on that surface fast enough to be useful and consistently enough to be trusted.
The two hard problems are not collection speed. They are property identity and stable coverage.
Model the listing lifecycle as events
Storing a nightly snapshot of every listing tells you what is on the market. It does not tell you what happened, and what happened is the signal.
Derive events from consecutive observations and store them as their own records:
| Event | Why it matters |
|---|---|
| New listing | New supply, the earliest leading indicator in the chain |
| Price reduction or increase | Depth and frequency are the clearest read on local momentum |
| Status change to under offer or sold subject to contract | Demand, and far earlier than the registry |
| Withdrawal | Often a soft-market signal, and easily confused with a sale |
| Relisting | Frequently the same property returning, which is where data integrity goes wrong |
| Sold, from the public register | The authoritative outcome, arriving late |
Each event needs the observation that opened it, the observation that closed it, and enough confidence metadata to say whether the boundary was observed or merely bounded by your sampling interval. If you collect daily, you know a price changed within a day, not when it changed.
Property identity is the whole system
The same property appears on several portals, under different reference numbers, with addresses formatted differently, sometimes with the unit designator in a different field or missing entirely. Over time it appears again, months later, as a fresh listing.
If identity resolution fails, two things break at once. Inventory inflates, because one property counts as three. And days-on-market collapses, because a relisted property looks new.
That second one deserves particular attention, because relisting to reset days-on-market is a deliberate practice in some markets. A feed that treats every relisting as new supply will systematically understate how long stock is sitting, in exactly the markets where that number matters most.
A workable identity strategy, in order of reliability: a parcel or title identifier where the jurisdiction exposes one; a normalised address plus unit designator, geocoded to a coordinate with a tight tolerance; then attribute matching on floor area, bedroom count and property type to break ties. Keep every portal’s own reference linked to your canonical property, and keep a listing-episode record so a property can have several episodes over time without losing its history.
Measure your match rate by hand-reviewing a sample, and publish it. An inventory figure without a known dedupe error rate is a number nobody can size.
Coverage stability beats coverage breadth
A housing index is read as a time series, which means any change in what you can see registers as a change in the market.
Add a portal mid-series and supply appears to jump. Lose one to a block and supply appears to fall. Start collecting deeper pages and inventory rises. None of that is the housing market, and all of it looks exactly like it.
So fix the panel before you start: a defined set of portals, a defined geography, a defined query set, and a defined depth per query, each collected fully or explicitly marked incomplete. Version the panel, stamp the version on every row, and treat any change to it as a documented methodology break rather than a quiet improvement.
This is the same discipline that governs any longitudinal web panel, and the fuller argument is in job-board data and labour-market intelligence. It transfers directly.
Geography is part of the observation
Portals localise. Search results, the properties surfaced, and sometimes the fields shown depend on where the request appears to come from, and portals serving several countries may return a different site entirely.
For a feed that reports regional markets, the collection has to come from the regions it reports on. With the Shifter gateway, the vantage point and session go in the credentials against p.shifter.io:443:
customer-USERNAME-country-gb-city-manchester-sid-feed-mcr-04-ttl-600:PASSWORDcountry-gb uses the ISO alpha-2 code, city-manchester narrows to the market, sid-feed-mcr-04 holds one exit across a full search including its pagination so the result set is internally coherent, and ttl-600 keeps that address for ten minutes. One session per query rather than per request is what stops page four belonging to a different vantage point than page one.
Keep locale signals consistent with the exit, since a mismatch changes what some portals return; that is covered in matching proxy geo, timezone and locale. Keep request rates ordinary with real backoff, as in rate limiting and request throttling.
Cadence by series, not one global setting
Daily collection is enough for almost everything on the listing surface, because listings do not change hourly and a daily pass gives you next-day event detection.
Faster is justified in two cases: a small watchlist in a hot market where under-offer status changes within hours, and the launch window of new-build releases. Public register data updates on its own schedule and collecting it more often than it publishes only produces duplicates.
What matters more than frequency is regularity. Collect at the same time each day, because a pass that drifts from morning to evening moves the event boundaries you are measuring.
The metrics that fall out
With events and identity in place, the outputs are straightforward, and each one needs its caveat stated in the product.
New supply and inventory, both dependent on dedupe quality. Days on market, with relisting policy disclosed, since the number changes materially depending on whether episodes are chained. Price-change frequency and depth, the cleanest momentum read available from listings. Withdrawal rate, useful but easily confused with sales. Asking-price indices, which are not transaction-price indices and should never be labelled as though they were. The distinction between observed, assessed, asked and estimated figures is set out in scraping valuations, rents and mortgage data. List-to-sale ratio and absorption, which need the register data and therefore lag.
Two publishing rules keep the feed credible. Publish the coverage metric beside the index, so a reader can tell a market move from a collection move. And suppress small cells: a median computed on nine listings in a postcode is noise with a decimal point, and it is the number most likely to be screenshotted.
The freshness contract
Every row should carry its observation time, and every consumer should apply its own staleness threshold rather than trusting that the table is current. That single design decision is what stops a delayed crawl from silently becoming a reported market change. It is the same contract described in building a real-time competitive price feed, and it applies unchanged here.
Track your own collection alongside the data: success rate per portal per market, and results returned against results expected. When supply appears to drop, the first question is whether the market changed or your coverage did. The method for establishing that baseline is in testing proxy speed, success rate and location accuracy.
FAQ
How real-time can housing data actually be?
Next-day event detection from a daily pass is achievable and sufficient for nearly every use. Intraday is worth it only for a small watchlist. The transaction record will always lag by weeks or months regardless of how you collect.
Why does our inventory count exceed the portals’ own numbers?
Almost always cross-portal duplication. One property listed with three agents is one property. Measure your match rate before trusting any inventory figure.
Should relistings count as new supply?
Pick a rule, document it, and apply it across the whole history. Chaining episodes to one property is usually more honest, since it preserves true time on market.
Can we publish an asking-price index as a house price index?
No. Asking prices lead and diverge from transaction prices, particularly when the market turns. Label it for what it is and the index is genuinely useful.
The bottom line
A real-time housing feed is an event pipeline over a slow market. Speed is the easy part. What determines whether anyone can rely on it is resolving properties to a stable identity, holding the panel fixed so coverage changes cannot masquerade as market moves, and labelling asking-price signals honestly against transaction data that arrives late.
Collect each market from that market, hold sessions coherent across a result set, attach an observation time to every row, and publish your coverage next to your index. The product view is on the data gathering pipelines page, with rates on the pricing page.