Real estate data collection means pulling structured information about properties from portals and databases: price, status, size, location, agent and the dates each of those changed. The technical difficulty is that the portals holding this data treat it as their core asset and defend it accordingly.
That makes proxy choice consequential. This guide covers what to collect, how to collect it from the major portals without being blocked, and what request rates and refresh cycles actually work.
What Setup You Need
For public listing pages, rotating residential proxies. The major portals run commercial anti-bot systems that classify datacenter address ranges before looking at anything else, so datacenter proxies fail on the first request regardless of rate.
For anything behind a login, including most MLS access, ISP proxies, because the session has to survive and a changing address will invalidate it.
Everything else is request rate discipline, which matters more here than the choice of proxy.
What to Collect
| Field | Why it matters | Refresh |
|---|---|---|
| Listing ID and URL | The key everything else joins on | Once |
| Price and price history | The primary signal in almost every analysis | Daily |
| Status (active, pending, sold, withdrawn) | Where market velocity comes from | Daily |
| Listed and updated dates | Days-on-market, the core health metric | Daily |
| Address, coordinates, postcode | Geographic aggregation | Once |
| Beds, baths, floor area, lot size | Normalising price into comparables | Once |
| Property type and year built | Segmentation | Once |
| Agent and brokerage | Market share analysis, and personal data, so treat carefully | Once |
| Description and features | Text analysis, amenity extraction | Weekly |
| Photo count and URLs | Listing quality signals | Weekly |
The distinction between fields that change and fields that do not is what keeps volume manageable. Price and status need daily checks. Floor area does not change, and re-collecting it daily multiplies your bandwidth for nothing.
Monitoring MLS Databases
MLS data is the richest source and the most restricted. It is not a public website: access is granted to licensed participants under an agreement, and the sanctioned route is a data feed rather than a scraper.
- RESO Web API is the modern standard, a structured API returning JSON, with incremental sync so you request only records changed since your last poll. If you are entitled to MLS access, this is the correct integration.
- RETS is the older feed standard, still in service in places, being retired in favour of the above.
- Poll incrementally, not fully. Ask for changes since your last timestamp. A full refresh of a large MLS is enormous and unnecessary.
- Respect the licence. MLS agreements govern display, retention and redistribution. Breaching them risks the access itself, which is a bigger loss than any dataset.
Proxies matter less here than for public portals, because you are an authenticated participant. Where they do help is stability: an ISP proxy gives the integration one fixed address, which matters when access is tied to registered addresses.
Gathering Data from the Major Portals
Public portals are where proxies do the work. Each behaves differently.
Zillow
Heavy anti-bot protection and aggressive rate limiting. Search result pages are the efficient entry point since they carry most summary fields without visiting each listing.
- Proxy: rotating residential, US geo-targeted.
- Rate: conservative. One request every 3 to 5 seconds per address.
- Watch for: map-based search returning data through internal endpoints rather than the HTML, which is both more efficient and more closely watched.
Redfin
Somewhat more tolerant than Zillow, and better structured. Data is often available in embedded JSON rather than requiring HTML parsing.
- Proxy: rotating residential, US.
- Rate: one request every 2 to 3 seconds per address.
- Watch for: coverage varies by market, so absence of a listing is not evidence it does not exist.
Realtor.com
Direct MLS syndication, so the data is timely, with moderate protection.
- Proxy: rotating residential, US.
- Rate: one request every 2 to 4 seconds per address.
Rightmove and Zoopla
The two dominant UK portals. Both defend themselves, Rightmove the more strictly of the two.
- Proxy: rotating residential, UK geo-targeted. Non-UK addresses see different results or none.
- Rate: one request every 3 to 5 seconds per address.
- Watch for: both list the same properties through different agents, so deduplicate on address rather than on listing ID.
Request Rates, Rotation and Concurrency
Rate is the single biggest factor in whether collection survives. Most people who get blocked were not detected as a proxy, they were detected as being in a hurry.
- Per-address rate: one request every 2 to 5 seconds on portals. Slower than feels necessary, and it is why concurrency exists.
- Concurrency, not speed: to collect faster, add addresses rather than shortening the interval. 10 addresses at one request every 3 seconds gives roughly 3 requests a second in total, which clears 10,000 listings in about an hour.
- Rotation: per request for browsing independent listing pages. Sticky sessions for paginating a search that carries a cursor, held just long enough to finish the pagination.
- Add jitter. Requests at exactly 3.0 second intervals are a signature. Vary between 2 and 5 seconds.
- Collect off-peak in the target market’s timezone. Lower baseline traffic means your requests are a smaller fraction of the total, and rate limits are usually looser.
When blocks start
- Read the response. A 429 is a rate limit and means slow down. A challenge page means your fingerprint is the problem, not your rate.
- Halve the rate first. It is the cheapest fix and usually the correct one.
- Check headers and fingerprint. A bare
User-Agenton an otherwise empty request is an obvious bot signal. - Do not retry into a block. Exponential backoff. Hammering a soft rate limit turns it into a hard ban on the address.
- Reconsider the target. If a portal costs more in retries than the data returns, the Web Scraping API handles rendering and challenges for you at a higher unit price but far lower maintenance.
Refresh Cadence
- Daily: new listings, status changes, price changes. This is where nearly all analytical value sits.
- Weekly: descriptions, photos, agent details. These change rarely after a listing goes live.
- Once: address, coordinates, size, property type. Structural facts.
Splitting the schedule this way typically cuts bandwidth by more than half against re-collecting everything daily, with no loss of signal.
Conclusion
Real estate data collection is mostly a discipline problem rather than a technical one. Use residential proxies because portals classify datacenter addresses on sight, keep per-address request rates slow and add concurrency instead, and split your refresh schedule by how often each field actually changes.
For the mechanics of scraping in general see web scraping proxies, for the commercial application price intelligence, and for the legal ground proxy and scraping legality. Shifter’s residential proxies cover the geo-targeting these portals require, with rates on the pricing page.