How to Tell When a Site Is Serving You Fake or Blocked Content

The scraping failure everyone plans for is the obvious one: a 403, a 429, a connection that times out. You can see it, count it, and retry it. The failure that actually poisons your dataset is the opposite: a 200 OK that looks completely successful and contains nothing you wanted. A block page dressed as a normal response. A challenge screen. An empty shell. A page of data that was fabricated specifically to feed a bot bad information.

A scraper that errors is annoying. A scraper that “succeeds” on garbage is dangerous, because you do not find out until the bad data is already in your warehouse, in a report, or in a model. This is the other half of why scrapers get blocked: modern anti-bot systems increasingly prefer to deceive quietly rather than reject loudly, precisely because a silent failure is worth more to them than a visible one. Here is how to catch it.

The status code is not the truth

The first habit to break is trusting HTTP status codes as a success signal. A 200 means the server sent a response, not that it sent the response you asked for. Sites routinely return block pages, CAPTCHAs, and consent walls with a 200, because doing so keeps naive scrapers happy and quiet while serving them nothing. Treat the status code as one weak signal and validate the body on every single request.

The categories of deceptive 200 are worth naming, because each has a different tell:

• Soft blocks and interstitials. A challenge or “verify you are human” page returned with a 200, often much smaller than a real page.
• Challenge and CAPTCHA screens. Served inline where the content should be. Related to how anti-bot defenses evolved.
• Degraded or stripped content. A login wall, an “enable JavaScript” stub, or an empty skeleton where the data was supposed to render.
• Rate-limit soft-fails. Stale, cached, or empty results returned instead of an error once you cross a threshold.
• Wrong geo or personalization. The right page for the wrong country, currency, or logged-out state, technically valid, silently useless.
• Honeypots and poisoned data. Content served deliberately to bots: fake prices, trap links, or plausible-looking records with impossible values.

Validate content, not just delivery

The single most effective defense is a content assertion on every response: a cheap check that the page contains what a real page must contain. Pick an invariant that a genuine result always has and a block page never does, a specific element, a required field, a minimum plausible length, and fail the request if it is missing.

def is_valid_product_page(html, parsed):
    # A real product page always has these. A block page has none of them.
    if len(html) < 2000:                      # block pages are usually tiny
        return False
    if parsed.select_one("h1.product-title") is None:
        return False                          # the anchor element is gone
    if parsed.select_one('[data-price]') is None:
        return False                          # the field we came for is missing
    return True

The key shift is that a missing anchor element is a failure, not an empty result. If the selector you rely on is absent, do not record a blank row and move on, treat the response as a soft block, and retry it with a fresh identity the same way you would a 403. Recording blanks is how a block quietly becomes thousands of empty rows nobody notices until much later.

Watch the shape of your responses, not just individual ones

Individual checks catch obvious junk. Distributional checks catch the subtle drift, and they are what separate a robust pipeline from a fragile one.

• Response size. Block and challenge pages tend to be small and uniform. A sudden collapse in average page size across a batch, or a spike in responses clustered at one exact byte count, is a block signature even if each one returns 200.
• Response hashing. Hash a normalized version of each response body. When the same hash suddenly repeats across many different URLs, you are being served one block page over and over, not real, varied content.
• Field fill rate. Track the percentage of records where each field actually populated. A field that was 98 percent filled yesterday and 4 percent filled today did not get less common, you started getting stripped pages.
• Success rate per host. A drop on one specific target, while others hold steady, is that target changing its posture toward you, worth catching before it wastes a whole run. This is the same per-target signal that matters when monitoring a scraping pipeline at scale.

None of these need machine learning. They are counters and simple baselines, and they are the difference between noticing a block in the first hundred requests and noticing it after a million.

Known block-page signatures

Maintain a small library of phrases and markers that only ever appear on block, challenge, or error pages, and flag any response that contains them regardless of status code.

BLOCK_MARKERS = (
    "verify you are human",
    "unusual traffic",
    "access denied",
    "enable javascript to continue",
    "request blocked",
)

def looks_blocked(text):
    low = text.lower()
    return any(marker in low for marker in BLOCK_MARKERS)

Keep it short and specific so it does not false-positive on real content that happens to discuss those topics, and grow it as you meet new defenses. A challenge page you can recognize is a challenge you can retry through instead of storing.

Honeypots and poisoned data

The nastiest category is content designed to look real. Two defenses matter here.

First, do not follow trap links. Honeypot links are commonly hidden from humans with display:none, visibility:hidden, zero size, off-screen positioning, or aria-hidden, and exist only to catch a bot that follows every anchor. A crawler that respects visibility, and ignores links a human could never click, sidesteps most of them.

Second, sanity-check the data itself. Poisoned records are built to pass a naive parser but tend to fail basic domain rules: prices that are zero or absurdly high, dates in the future, quantities that cannot exist, values outside any plausible range. Validate against the ranges your domain actually permits, and quarantine records that violate them rather than trusting them because the HTML parsed cleanly.

Canary requests

The most reliable early warning is a canary: periodically fetch a page whose correct content you already know, and assert that it still matches. When your canary starts returning a block page or the wrong data, you know the target flipped on you, immediately and unambiguously, without having to infer it from a slow decline in data quality. Run canaries per target and per geo, since a site can block one country’s exit IPs while leaving another untouched.

Where proxies fit

Clean IPs reduce how often you are soft-blocked in the first place, a pool with good reputation sails through where flagged addresses get quietly fed a challenge page. That lowers the rate of deceptive responses you have to catch, but it does not remove the need to catch them, because blocks are increasingly invisible by design and no IP is immune. The two work together: detection tells you a response was a soft block, and a rotating residential pool gives you a fresh identity to retry it with, exactly as you would treat a hard failure. Feed every detected soft block back into your retry and rotation logic (sticky vs rotating), and treat a persistent per-target block-rate spike as a signal to back off rather than to hammer (avoiding blocks and scraping responsibly both apply).

The bottom line

Assume the response is lying until it proves otherwise. Validate content on every request against an invariant a real page always has, and treat a missing anchor as a failure to retry, not an empty row to store. Watch the distribution of response sizes, hashes, and fill rates so a block that returns 200 still shows up as an anomaly. Keep a short signature list for known block pages, refuse to follow hidden trap links, sanity-check data against domain rules, and run canaries so you learn the instant a target turns on you.

Do that, and the silent failure, the one that quietly corrupts a dataset for weeks, stops being silent. Pair it with a clean residential pool so you are soft-blocked less often to begin with, and the per-GB plans let you validate this against your own targets without committing up front.