Assortment analysis sounds like set arithmetic. Take their catalog, take yours, subtract. The gaps are the opportunity.
The arithmetic is trivial. Everything that makes the answer right or wrong happens before it: whether you enumerated their catalog or merely sampled it, and whether “the same product” on their site and yours is actually recognized as the same product. Get either wrong and the subtraction produces a confident list of opportunities that are mostly artifacts.
The matching problem is the whole problem
The same item is described differently by every retailer that sells it. Different title conventions, different brand spellings, different pack sizes, different bundling, and internal SKUs that mean nothing outside each company.
If your matcher fails on twenty percent of items, then twenty percent of their catalog looks like a gap you should fill and twenty percent of yours looks like unique differentiation. Both conclusions are wrong, they are wrong in the direction that makes the analysis look valuable, and nothing in the output flags them.
Match in this order:
Standard identifiers first. GTIN, UPC, EAN, ISBN, MPN. Where these are published they are close to definitive, and they should be your primary key wherever available.
Brand plus normalized model second. Normalize aggressively: case, punctuation, whitespace, brand aliases, common abbreviations. Store the normalized form so a matching decision can be audited later.
Attribute matching third. Size, colour, capacity, pack count. This is where pack size does the most damage, because a six-pack and a single are genuinely different products that share almost every attribute and often share a title.
Human review of a sample, always. Pull a random sample of matches and non-matches and have someone check them. This produces the number that makes everything else interpretable: your match rate. An analysis reported without it is an analysis nobody can size the error on.
Enumerate, do not sample
The second failure is treating a crawl as a census when it was a sample.
Most catalogue surfaces cap what they will show you. A category listing may stop at a fixed number of pages regardless of how many products it claims. Search results are usually capped harder than category traversal. Filters and facets are frequently the only way to reach the tail, by slicing a large category into segments each small enough to be fully returned.
The practical method is to traverse the category tree rather than searching, subdivide any node whose result count exceeds the visible page cap, and record for every node both the claimed product count and the number you actually retrieved. That second number is your evidence. When claimed and retrieved diverge, you have found the boundary of your own visibility, and everything beyond it will present as a gap in your catalogue rather than a gap in your data.
Category structures also differ between retailers, so do not assume their category is your category. Match at the product level and roll up to your own taxonomy afterwards.
The retailer-specific mechanics are in scraping Amazon product data and monitoring product availability and stock.
Catalogs are regional
The same retailer carries different assortments by market, and often serves a different catalogue entirely by storefront.
If your collection runs from one country, every product they carry only in another market reads as absent, and every product carried only in the market you happen to collect from reads as universal. For a business selling across markets, that is not a small correction. It changes which gaps are real.
Residential proxies with country targeting keep each market’s catalogue distinct. With the Shifter gateway, targeting and session go in the credentials against p.shifter.io:443:
customer-USERNAME-country-es-sid-cat-es-0119-ttl-600:PASSWORDcountry-es sets the market, sid-cat-es-0119 holds one exit across a full category traversal so pagination stays coherent, and ttl-600 keeps that address for ten minutes. One session per category walk, not one per page request, is the rule. Rotating mid-traversal is how pages from two different vantage points end up in one catalogue snapshot.
Keep concurrency modest and back off on errors rather than pushing through them, as in rate limiting and request throttling. The product view is on the residential proxies for e-commerce page.
The four outputs
Once products are matched and catalogs enumerated, the comparison yields four distinct things, and conflating them is the most common analytical error.
True gaps. They carry it, you do not. The headline output, and only trustworthy in proportion to your match rate.
Reverse gaps. You carry it, they do not. Usually ignored, and usually more valuable, because this is your differentiation inventory and the merchandising team can act on it immediately.
Depth gaps. You both carry the brand, they carry forty SKUs and you carry six. This rarely appears in a naive set difference because the brand is present on both sides, and it is often the biggest commercial finding in the whole exercise.
Phantom gaps. Matching failures wearing the costume of true gaps. Every analysis has them. The question is whether you measured how many.
Report the first three with the fourth quantified alongside. “Two hundred and forty true gaps at a ninety-one percent match rate” is a usable finding. “Two hundred and forty gaps” is a number waiting to waste a category manager’s week.
Cadence and drift
Catalogs move much more slowly than prices, which is convenient: a weekly or fortnightly full enumeration is usually enough, with a faster cadence on categories under active competition.
What matters more than frequency is stability. Keep the traversal, the matching rules and the category mapping fixed between runs, and version them. If your gap count drops sharply, the first question is whether their assortment changed or your matcher did. A run-over-run diff of match rate and retrieved counts answers that in seconds and prevents a lot of wasted analysis.
FAQ
What match rate is good enough to act on?
Above ninety percent is workable for category-level decisions. Below eighty, the gap list is mostly matching noise and the effort belongs in the matcher rather than in the analysis.
Should I use search or category traversal to enumerate?
Category traversal, subdivided with facets where a node exceeds the page cap. Search is capped more aggressively and its ranking is personalized, which makes it a poor enumeration tool.
How do I handle marketplace listings from third-party sellers?
Track them as a separate set. Blending first-party and marketplace assortment overstates a competitor’s committed range, since a third-party listing is not a buying decision they made.
What if they publish no standard identifiers?
Lean harder on normalized brand and model, accept a lower match rate, and measure it honestly. Some categories, particularly apparel and private label, will never match cleanly, and the right response is to report the confidence rather than the certainty.
The bottom line
The comparison at the end of assortment analysis is arithmetic. The work is upstream: enumerating a catalog rather than sampling it, knowing where your visibility stops, matching products with a rule you can audit, and separating true gaps from depth gaps and from matching failures.
Collect each market separately, hold sessions coherent across a traversal, record claimed counts against retrieved counts, and publish your match rate beside every gap number. Rates are on the pricing page.