Collecting listings from one portal is a scraping problem, and a well understood one. Collecting them from thirty portals across a dozen countries and presenting the result as one searchable inventory is a different job entirely, and almost none of the difficulty is in the fetching.
It is in the fact that two portals describing the same apartment will disagree about its size, its room count, its price and even what kind of property it is, and all of them will be right by their own local convention.
If you are still setting up the collection itself, proxies for real estate data covers that ground. This guide is about what happens after the data arrives.
The fields that do not mean what they say
Five categories cause most cross-portal aggregation failures.
Area. Square metres, square feet, and in some markets local units. Worse, the measurement basis differs: gross internal, net internal, and in several countries a legally defined measurement standard that excludes or discounts parts of a property. Converting units is trivial; reconciling bases is not, and a straight conversion silently mixes them.
Room counts. In much of continental Europe the headline figure counts rooms including living rooms rather than bedrooms. A “3 pièces” and a “3 bedroom” are not the same property. Storing both in a single bedrooms column produces an inventory that is wrong in a way that only shows up when someone compares markets.
Price semantics. Asking price, guide price, “offers over”, auction reserve, price on application, and in rental markets whether the figure includes service charges, utilities or local taxes. These are different quantities wearing the same currency symbol.
Tenure and ownership type. Freehold, leasehold with a remaining term, strata or condominium arrangements, cooperative ownership. A leasehold with a short unexpired term is a materially different asset from a freehold at the same price.
Property type taxonomies. Every portal has its own, and they do not map cleanly. Deciding whether a maisonette, a duplex and an apartment with internal stairs are one category or three is a product decision that has to be made once and applied everywhere.
The principle that keeps this manageable: store the source’s original values verbatim, alongside your normalised values, always. When you later discover that one portal’s area basis differs from what you assumed, the raw fields let you re-derive. Without them you have to re-collect, and historical data is simply lost.
Build the canonical model before the second portal
The order matters. Teams that integrate portal one, then bolt portal two onto its schema, end up with a canonical model that is really portal one’s model wearing a different name, and every subsequent integration fights it.
A workable canonical record separates three layers:
| Layer | Contents | Why separate |
|---|---|---|
| Raw | The source’s fields exactly as published, plus the fetch metadata | Lets you re-derive when your assumptions change |
| Normalised | Your units, your taxonomy, your price semantics, with the conversion recorded | What the product queries |
| Derived | Price per unit area, computed indices, scores | Recomputable, never the source of truth |
Every normalised field should carry a note of which rule produced it. When a client asks why a property shows 68 square metres on your platform and 73 on the portal, the answer should be a lookup rather than an investigation.
Currency, and never storing only the converted value
For multi-country inventories, store the original amount and its currency code as published, plus the exchange rate you applied and the date of that rate.
Storing only a converted figure destroys information irreversibly. Rates move, corrections get issued, and a client viewing a historical listing wants the price that was asked, not that price re-expressed at today’s rate. Convert at query time from the stored original, or store the conversion with its dated rate so it can be audited and redone.
Reconciling the same property from several portals
The same property routinely appears on several portals, listed by different agents, with different photographs, different descriptions and sometimes different prices. Identity resolution is what turns that into one record, and it is covered in depth in building a real-time housing market data feed, since the same machinery drives inventory counting there.
What is specific to aggregation is what you do once the duplicates are grouped: deciding which value wins.
Define a source precedence per field rather than per portal. One portal may have the most reliable area figures while another has better photographs and a third updates prices fastest. A single global ranking throws that away.
Then handle disagreement explicitly. When grouped listings disagree on price beyond a tolerance, that is a signal rather than an error: it can mean a price change one agent has not reflected, or a mismatched grouping. Flag it, show the range, and keep the alternatives linked. Silently picking one and discarding the rest is how an aggregator loses trust.
The general matching discipline, including measuring and publishing your match rate, is the same one described in competitor assortment and catalog gaps.
Coverage is national, not global
There is no global real estate market and no global portal set. Each country has its own leading portals, its own agent behaviour, and its own conventions about what gets listed publicly at all. In some markets a large share of transactions never appears on a public portal.
So treat a global inventory as a union of national panels, each with its own defined portal set, and record coverage per market rather than in aggregate. A single headline count across countries hides the market where you have one portal and the market where you have six.
Two practical rules. Do not compare absolute inventory across markets unless coverage is comparable, because you will be measuring your own panel. And where licensed feeds exist for a market, take the licence: coverage and field quality are usually far better than public collection, and the legal position is simpler.
The collection layer
Portals localise heavily. What you see, the currency displayed, the language, sometimes the site itself, depends on where the request appears to come from. A multi-country aggregator collecting from one vantage point will quietly receive one country’s view of several markets.
With the Shifter gateway, the market and session go in the credentials against p.shifter.io:443:
customer-USERNAME-country-fr-sid-listings-fr-12-ttl-600:PASSWORDcountry-fr uses the ISO alpha-2 code, sid-listings-fr-12 holds one exit across a full search including pagination so a result set is internally coherent, and ttl-600 keeps that address for ten minutes. Without sid the gateway rotates per request, which is right for independent lookups and wrong for a paginated search.
Keep locale signals consistent with the exit, since a mismatch changes what some portals return, and keep request rates ordinary with real backoff, as in rate limiting and request throttling. Track collection success per portal per market alongside the listings, because a portal that quietly starts returning fewer results looks exactly like a market with less supply.
Field completeness is a quality metric, not a detail
Portals differ enormously in how completely they populate optional fields. An aggregator that treats a missing field as absent rather than unpublished will report, for example, that one market has almost no properties with energy ratings when in fact one portal simply does not expose them.
Score completeness per portal per field, publish it internally, and use it when choosing source precedence. It also tells you where a licensed feed would actually improve the product rather than just cost money.
FAQ
Should I normalise on ingestion or at query time?
Normalise on ingestion and keep the raw fields. Query-time normalisation is slower and makes indexing painful, but without the raw values you cannot fix a bad rule retroactively.
How do I handle portals that publish rooms rather than bedrooms?
Store both concepts as separate fields and populate what the source gives. Do not infer bedrooms from a room count, and do not let a product filter query a field that is only populated in some markets.
Is one canonical property type taxonomy realistic across countries?
A shallow one is. Keep the top level small and portable, and put local specificity in a secondary field rather than forcing it into the main taxonomy.
What is the highest-value thing to fix first?
Area basis and price semantics. They affect every derived metric, and errors in them are invisible until someone compares two markets.
The bottom line
Cross-portal aggregation is a normalisation problem wearing a scraping problem’s clothes. The fetching is the part that is already solved.
Build the canonical model before the second integration, keep the source’s raw values forever, store original currency with a dated rate, set source precedence per field rather than per portal, treat disagreement as a signal rather than an error, and record coverage per market because there is no global panel. The product view is on the large-scale data gathering page, with rates on the pricing page.