Real estate teams do not want to run scraping infrastructure. They want to know where supply is rising, where prices are being cut, what comparable properties are asking and how long stock is sitting. The collection layer is a means to that, and for many teams a web scraping API is the most direct way to get it without hiring for browser automation and proxy operations.
What to collect and how to model it once collected is covered elsewhere on this blog: valuations, rents and mortgage data, a real-time housing market feed, and aggregating listings across portals. This piece is about the collection layer itself: why property portals push teams toward an API, how the API maps onto real estate pages, and how the economics work.
Why property portals are awkward to collect
Four properties of real estate sites make them harder than most.
They are built for maps and scrolling, not pages. Search results load through JavaScript as a map moves or a list scrolls, so a plain HTTP request often returns an empty shell.
Results are paginated and cursor-driven. Moving from page one to page two frequently depends on server-side state, which breaks naive page-by-page fetching.
They are localised. Portals serve by country and sometimes region, so what you see depends on where the request appears to come from.
They are defended. High-value, frequently refreshed inventory attracts automated traffic, and portals protect it accordingly.
Handling all four in-house means headless browsers, proxy management, retry logic and fingerprint tuning. A web scraping API bundles those into one request.
How the API maps onto real estate pages
With the Shifter Web Scraping API, each portal problem has a direct control.
Map and list views. render_js=1 runs the page in headless Chrome at no extra credit cost, and wait_for_css holds the capture until the listing cards have actually rendered. JavaScript instructions can scroll or click a “load more” control before capture. See rendering JavaScript.
Result cards. extract_rules with a list type returns every listing card on a page as JSON, one object per listing, without an HTML parser on your side.
Pagination. session_id holds cookies, browser state and the upstream IP across requests, so a cursor-driven result set can be walked in order. Sessions expire after 10 minutes idle, and the country should stay the same for the life of a session. See sessions and proxies.
Localisation. country takes an ISO alpha-2 code per request. Global geolocation and the residential pool via premium_proxy=1 are available on Growth plans and above.
Evidence. screenshot=1 captures the rendered page, useful when a listing’s condition at a point in time matters, such as a price cut or a withdrawal.
Long renders. webhook=<URL> delivers the response to your endpoint when it is ready instead of holding a connection open.
A search results request for one market looks like this:
curl "https://scrape.shifter.io/v1?api_key=YOUR_API_KEY\
&url=https%3A%2F%2Fportal.example.com%2Fsearch%3Fcity%3Dlyon\
&render_js=1&wait_for_css=.listing-card\
&country=fr&premium_proxy=1&session_id=lyon-walk-03\
&extract_rules=%7B%22listings%22%3A%7B%22selector%22%3A%22.listing-card%22%2C%22type%22%3A%22list%22%2C%22item%22%3A%7B%22price%22%3A%7B%22selector%22%3A%22.price%22%2C%22output%22%3A%22text%22%7D%2C%22area%22%3A%7B%22selector%22%3A%22.area%22%2C%22output%22%3A%22text%22%7D%2C%22link%22%3A%7B%22selector%22%3A%22a%22%2C%22output%22%3A%22%40href%22%7D%7D%7D%7D"The target URL is URL-encoded because it carries its own query string, which would otherwise be read as parameters of the API request. The response is one JSON object with a listings array. A field whose selector finds nothing comes back as null, which matters for monitoring, as covered below.
How different teams use it
Acquisitions and investment teams watch target submarkets for new supply and price reductions, and use the timing of cuts as a negotiation signal. What they need is next-day event detection on a defined set of areas, not a national crawl.
Brokerages track their share of listings against competitors by area, and how quickly competing listings move. Keep this at the brokerage level: agent contact details in listings are personal data and are rarely what the analysis needs.
PropTech products build comparables and listing feeds for their own users. Here the API is one input to a normalisation pipeline, and field definitions differ by portal and country.
Rental operators monitor asking rents and concessions in their submarkets. Concessions often sit in description text rather than a price field, so extraction rules should capture the description, not just the headline rent.
Lenders and insurers follow market conditions in the areas where they hold exposure. The boundary is published market data, never individual borrower or occupant information.
The economics: design around the credit
One credit buys one successful request, whatever that request returns. Rendering, extraction rules, screenshots and the API’s own retries are included, and failed requests or target errors are not charged.
That has a direct design consequence for real estate. A search results page that returns forty listing cards costs the same single credit as a detail page that returns one listing. So the efficient pattern is to collect from result pages wherever the card carries the fields you need, price, area, rooms, link, and fetch a detail page only when a listing is new or its card has changed.
For a market with a few thousand active listings, that difference is often an order of magnitude in credits. It also improves freshness, because you can revisit result pages more often for the same spend.
Two more cost levers. Refresh on a cadence that matches how fast each market moves, since daily is enough for most listing surfaces. And keep an eye on credits spent against rows that parsed cleanly, because a broken selector still consumes a credit for every successful but useless response.
Monitor for silent breakage
Portals redesign, and a changed selector does not fail the request. It returns null, the request succeeds and the credit is spent.
Track the null rate per field, per portal and per country over a rolling window, and alert when it jumps from its baseline. Version your extraction rules so a fix can be traced, and land raw responses before parsing so a corrected rule can be replayed without paying to fetch again. The loading pattern is set out in moving web scraping API data into SQL.
When an API is the right collection layer, and when it is not
Choose the API when rendering and anti-bot handling are the main burden, when the team is small or data-focused rather than infrastructure-focused, and when predictable per-success billing matters more than the lowest possible unit cost.
Choose self-managed proxies when you need fully custom browser flows, run at a scale where owning the stack is cheaper, or already have scraping engineers. The proxy-based approach is covered in proxies for real estate data.
Choose a licensed feed first wherever one exists for your market. Coverage and field quality are usually better, and the terms are clear. Use collection for what a licence does not provide.
Staying on the right side of it
Respect each portal’s terms and keep request volumes proportionate. Treat owner, agent and occupant details as personal data by default and strip them at ingestion where the analysis does not need them. Use screenshots as evidence of what a listing showed, not as material to republish. The broader framing is in residential proxies and GDPR compliance.
FAQ
Do I need JavaScript rendering for property portals?
For most modern portals, yes, because listing results load after the initial page. It costs the same credit as a static fetch, so there is little reason not to enable it where results are rendered client-side.
Can one API cover portals in several countries?
Yes, with country set per request and a session per market. Global geolocation requires a Growth plan or above.
How do I walk paginated search results reliably?
Use a session_id per search, keep the country constant within it, and keep the walk moving, since sessions expire after 10 minutes idle.
Is it cheaper to scrape detail pages or result pages?
Result pages, wherever the listing card carries the fields you need. One credit returns many listings, and detail pages can be reserved for new or changed listings.
The bottom line
For most real estate teams, the hard part of market intelligence is not deciding what to measure. It is getting reliable data out of portals built for maps, scrolling and human visitors. A web scraping API turns rendering, pagination, localisation and retries into request parameters, and bills only when data arrives.
Design around the credit by collecting from result pages first, walk searches with sessions per market, watch null rates for silent breakage, and take a licensed feed wherever one exists. The product is on the Web Scraping API page, with plans on the pricing page, and the wider use case on the real estate page.