PropTech teams tend to describe their data requirement as “property data”, as though valuations, rents and mortgage rates were three columns of one table. They are three different problems. They come from different sources, they carry different licensing and privacy constraints, and, most importantly, they mean different things.
The failures that follow from treating them as one dataset are not collection failures. They are labelling failures, and they surface much later, when a user disputes a number your product presented as a fact.
Three data types, three sourcing realities
Before writing a collector, be clear about which of these you are actually acquiring, because the answer determines both the method and what you may claim.
| Data type | Authoritative source | What the web gives you | What it is not |
|---|---|---|---|
| Transaction prices | Public records: land registry, deeds, assessor rolls | Sold prices, transfer dates, parcel attributes | Not current market value |
| Assessed values | Tax assessor rolls | An assessment for taxation | Not a market valuation |
| Portal estimates | The portal’s own model | A proprietary model output | Not an appraisal, not a fact |
| Asking rents | Active listings | What landlords are asking today | Not what tenants are paying |
| Achieved rents | Licensed datasets, operator data | Rarely public | Not derivable from listings |
| Mortgage rates | Lender rate pages, official statistical series | Published rates by product and tier | Not the rate any borrower got |
The single most useful discipline in PropTech data is keeping “observed”, “assessed”, “asked” and “estimated” in separate columns and never letting them collapse into one field called value.
Start with the sources that are meant to be used
Work down this order, and only reach the bottom for what the layers above genuinely do not cover.
Official and bulk sources first. Land registries, assessor offices and statistical agencies frequently publish bulk files or APIs. They are authoritative, they are permitted, and they carry history you cannot reconstruct externally. For transaction prices and parcel attributes this is usually the whole answer.
Licensed feeds second. Broker and multi-listing data is generally available under licence. Where your product needs full listing coverage with reliable fields, a licence is cheaper than the engineering and legal exposure of the alternative.
Public web observation last, for what the first two do not provide: asking rents and their concessions, live inventory, published lender rate tables, and how any of these differ by market.
Skipping straight to the bottom layer is the most common and most expensive mistake in this category. The listings side of that work is covered in proxies for real estate data.
Valuations: never present a model as a measurement
Portal estimates are model outputs trained on data you cannot see, with error distributions the portal may or may not publish. They are useful as a feature in your own model and they are not a valuation.
Three rules keep this defensible. Store the estimate with its source and the date it was observed, never as a bare number. Do not relabel it: an estimate is an estimate, not “market value”. And if your own product produces a valuation, publish a confidence interval alongside it, because a point estimate presented without one will be read as precision you do not have.
For anything approaching a formal valuation, the collected data is an input to a professional process, not a substitute for it.
Rents: asking is not achieved, and the concession is in the text
Listing rents are asking rents. In a soft market they run above what tenants actually pay, and the gap is where the concessions live: a month free, waived fees, a parking space included, a shorter term at a different price.
Those concessions are usually in the free-text description rather than in a structured field, which means an asking-rent series built from the rent field alone is systematically wrong in exactly the conditions people most want to measure. Parse the description for concession language and store it as its own field, even if all you can extract is a flag and the raw sentence.
Two more practicalities. Normalise to a comparable unit, typically rent per period per bedroom count or per floor area, since a median across mixed unit types measures the mix as much as the market. And keep unit-level and building-level observations separate, because a building listing one unit is not a building renting at that price.
Mortgage data: rates are a matrix, not a number
Published lender rates vary by product, term, loan-to-value band, borrower tier, region and sometimes channel. A scraper that stores “the rate” from a lender page has captured one cell of a matrix and thrown away the axes.
Record the product, the term, the LTV band, any stated borrower conditions, points or fees, whether the figure is a nominal rate or an APR, the effective date the lender states, and the date you observed it. The distinction between nominal and APR is the one that causes the most downstream confusion, because the two are not comparable and both appear on the same page.
Official statistical series from central banks and housing agencies are the ground truth to sanity-check your collection against. When your collected average diverges from the published series, the collection is usually what changed.
One boundary that is not negotiable: collect published rates and terms, never borrower-level data. Individual applications, credit files and personal financial details are not public-web data, and nothing in a PropTech roadmap justifies going after them.
The collection layer
Two properties of these sources make the vantage point part of the method.
Lender rate tables and portal content are regionalised, so the rate or the listing you see depends on where the request appears to come from. And rates in particular can differ by state or region within one country, which means a single national vantage point silently reports one region’s rates as the market.
With the Shifter gateway, the vantage point and session go in the credentials against p.shifter.io:443:
customer-USERNAME-country-us-state-tx-sid-rates-tx-11-ttl-600:PASSWORDcountry-us and state-tx place the request in the market whose rates you are reading, sid-rates-tx-11 holds one exit across a full rate-table walk so every cell in a snapshot comes from one session, and ttl-600 keeps that address for ten minutes. ttl only does anything alongside sid; without a session identifier the gateway rotates per request, which is right for independent lookups and wrong for a paginated table. The trade-off is set out in sticky vs rotating residential proxies.
Add strict-true when an exact regional match matters more than getting a response, so the gateway returns a 502 rather than quietly serving you a nearby region.
Cadence should follow how fast each series actually moves. Lender rates justify daily and sometimes intraday collection. Listings and asking rents are daily. Public records update on their own schedule, often weekly or monthly, and collecting them faster than they publish just produces duplicate rows. Keep request rates ordinary with real backoff, as in rate limiting and request throttling.
The schema that keeps you honest
Every record should carry, alongside the number itself: the observation type from the table above, the source, the effective date the source states, the date you observed it, the market and the exit location you observed from, and a confidence or validation flag.
Effective date and observed date are different fields and conflating them is a real bug. A lender rate sheet effective last Tuesday, collected today, is a Tuesday rate. An index built on observation dates will show a move that did not happen when your crawler was late.
The general argument for recording where and when an observation was made is in the case for a vantage-point standard.
Privacy and compliance
Property data sits closer to individuals than most commercial web data, and the rules vary sharply by jurisdiction.
Public records in some countries name owners; in others that information is restricted. Treat owner names, contact details and anything identifying an occupant as personal data by default, collect them only where you have a basis, and strip them at ingestion where your product does not need them. An index of rents does not need a tenant’s name.
Respect each source’s terms, keep volumes proportionate, and where a licence is the intended route, take the licence. The general framing is in residential proxies and GDPR compliance and ethical residential proxies for AI data collection.
FAQ
Can I use portal estimates as valuations in my product?
Not as valuations. As a feature, or as a clearly attributed third-party estimate, yes, subject to the portal’s terms. The labelling is the part that matters.
How do I get achieved rents rather than asking rents?
Generally through licensed datasets or operator partnerships. Listings do not contain them, and inferring them from asking rents is modelling, which should be disclosed as such.
Is scraping public records necessary if bulk files exist?
No, and bulk files are better: authoritative, complete and permitted. Use the web only for what is not published in bulk.
Why do our mortgage rates disagree with the published national average?
Usually because you collected one region, one LTV band or one borrower tier and averaged across an unrepresentative mix. Record the axes and the disagreement usually explains itself.
The bottom line
Valuations, rents and mortgage rates are three collection problems that share a vocabulary and nothing else. The teams that build reliable PropTech data go to official and licensed sources first, use the public web for the residue, and keep observed, assessed, asked and estimated as separate labelled fields with both an effective date and an observation date.
Collect each market from that market, keep sessions coherent across a table or a result set, and never let a model output be stored as a fact. Turning those observations into a live market series is covered in building a real-time housing market data feed. The product view is on the residential proxies for data gathering page, with rates on the pricing page.