Dynamic pricing gets discussed as an algorithm problem. Pick an elasticity model, set some guardrails, let it optimise.
In practice the algorithm is rarely what fails. The feed underneath it is. A repricer acting on a stale number, a mismatched product, or an observation collected from the wrong market will make a confident, fast, wrong decision, and it will keep making it every cycle until someone notices the margin.
So the useful way to think about a competitive price feed is not as a scraping job that fills a table. It is as a measurement system with a freshness guarantee, a confidence score, and a safety layer that decides when the pricing engine should be allowed to act on it at all.
What “real-time” actually needs to mean
Real-time is a marketing word. The operational question is narrower: how stale can an observation be before it stops being safe to price against?
That number is not universal. It falls out of how fast your competitors actually move, which you can measure rather than guess. In slow categories, a competitor’s price may hold for weeks and a daily feed is genuinely real-time enough. In contested categories with automated repricers on both sides, a four-hour-old observation can already be wrong.
Two things follow.
Tier your catalogue by volatility, not by revenue. The instinct is to refresh the top sellers most often. The right rule is to refresh what moves most often, which overlaps with top sellers but is not the same set. A slow-moving high-revenue item does not need hourly collection.
Store the observation time, and let the consumer decide. Every row in the feed carries when it was observed. The pricing engine applies its own staleness threshold rather than trusting that everything in the table is current. This one decision prevents most of the damage a delayed crawl can do.
The pipeline in order
match -> collect -> validate -> normalise -> score -> serveThe stages people skip are validate and score, and those are the two that make the feed safe.
Matching comes before everything
A price feed that compares the wrong products produces worse decisions than no feed at all, because it produces them with conviction.
Match on standard identifiers where they exist, GTIN, UPC, EAN or MPN, then on normalised brand and model, then on attributes. Pack size is the classic trap: a six-pack and a single share almost every attribute and often share a title, and pricing against the wrong one moves your price in exactly the wrong direction.
Every matched pair should carry a confidence score, and low-confidence pairs should be excluded from automated repricing even when they are good enough for a human dashboard. The fuller treatment of the matching problem is in competitor assortment and catalog gaps.
Collect the whole offer, not the number
The displayed price is one field in a larger offer, and pricing against it alone is how you end up undercutting a competitor who was never actually cheaper.
Capture the price, any strike-through or reference price, the delivery cost and threshold, stock state, seller identity, and any badge or promotional text. A competitor at the same price with free delivery is cheaper. A competitor who is out of stock is not a competitor this hour. A third-party marketplace seller is usually not the competitor your pricing committee meant.
Promotions deserve their own handling, because most of the mechanisms that change what a customer pays never touch the price field. That is covered in tracking competitor promotions and discount cycles.
Validate before the feed, not after
Every observation should pass a set of cheap checks before it is allowed into the table:
- Did the page render as a product page, rather than a challenge, a redirect or a soft 404?
- Did the request exit from the market this row claims?
- Is the price inside a plausible band relative to recent history for that item?
- Does the currency match the market?
- Is the product identifier on the page the one we requested?
The band check catches the most expensive class of error. A price that moves 60% overnight is occasionally real and usually a parsing failure, a currency mix-up or a bundle page. Flag it, hold it out of automated pricing, and let a human confirm.
Score confidence, then serve
The feed’s contract with the pricing engine should be explicit: here is the price, here is when it was observed, here is the market, here is how confident we are, and here is whether you may act on it automatically.
A simple three-state signal is enough for most teams. Act automatically, surface to a human, or ignore. Confidence combines match quality, observation age, validation results and recent volatility for that item.
Geography is part of the price
The same competitor page can show different prices, different delivery options and different stock in different markets. A feed collected from one vantage point and used to price several markets is quietly importing one country’s competitive landscape into all of them.
This is where the collection layer stops being an implementation detail. Residential proxies with country targeting let each market’s feed actually come from that market. With the Shifter gateway, targeting and session go in the credentials against p.shifter.io:443:
customer-USERNAME-country-de-sid-feed-de-08-ttl-600:PASSWORDcountry-de sets the market, sid-feed-de-08 holds one exit across a category walk so the prices inside a single snapshot are internally consistent, and ttl-600 keeps that address for ten minutes. Rotating mid-crawl is how two markets’ prices end up inside one snapshot.
Keep concurrency modest and back off on errors rather than pushing through them, as in rate limiting and request throttling. A feed that triggers defenses becomes a feed with gaps, and gaps in a pricing feed are not neutral: they are silently biased toward the items and hours that were hardest to collect.
The safety layer
Automated pricing needs brakes that sit outside the model, because a model is only as sane as its inputs.
Floors and ceilings per item. Cost-plus floors and a maximum discount, enforced after the model, not inside it.
Rate of change limits. Cap how far a price may move in one cycle and in one day. A correct-looking sequence of small moves driven by a bad feed is still a race to the bottom.
Coverage gates. If fewer than a defined share of tracked competitors returned a valid observation this cycle, do not reprice. Repricing against a thin sample is worse than holding.
A kill switch, and someone who owns it. Automated pricing failures compound quickly. The ability to freeze all automated moves in one action, with a named owner, is not optional.
Note that the coverage gate depends on measuring your own collection, not just the prices. Track success rate per competitor per cycle alongside the feed itself, using the method in testing proxy speed, success rate and location accuracy. When margin moves, the first question is whether the market changed or your coverage did.
What to monitor once it is live
A price feed degrades quietly. These four metrics surface it early:
| Metric | What it tells you |
|---|---|
| Freshness distribution | The share of the catalogue inside its staleness threshold, per market |
| Collection success rate | Per competitor and per market, so a single failing target is visible |
| Match confidence mix | Whether the share of low-confidence pairs is drifting upward |
| Validation rejection rate | A rising rate usually means a site changed its markup, not that prices went strange |
Alert on the deltas rather than the levels. A validation rejection rate that doubles overnight is a parser problem you want to hear about before the pricing engine acts on whatever got through.
FAQ
How often should a competitive price feed refresh?
Faster than your competitors move on the items that matter, which you should measure rather than assume. Tier the catalogue by observed volatility and spend the collection budget there.
Should marketplace third-party sellers be in the feed?
Track them, but keep them as a separate series and decide deliberately whether they enter the pricing logic. Blending first-party and marketplace offers produces a competitive picture nobody actually competes against.
What is the most common cause of a bad automated price?
Product mismatch, followed by stale observations treated as current. Both are prevented by fields the feed should already carry: match confidence and observation timestamp.
Do I need a separate feed per country?
Per market, yes, collected from that market. One feed reused across markets will be right in the country you collected from and progressively wrong elsewhere.
The bottom line
A dynamic pricing system is a measurement system with a decision layer attached, and most of its failures live in the measurement half. Match before you collect, capture the whole offer rather than the number, validate every observation before it lands, attach freshness and confidence to every row, and let the pricing engine refuse to act when the feed cannot support it.
Collect each market from that market, keep sessions coherent across a snapshot, and monitor your own coverage as closely as you monitor competitors. The product view of this work is on the residential proxies for price monitoring page, with rates on the pricing page.