The web’s defences were built around a simple split: people on one side, automation on the other. An AI agent is neither. It acts on a person’s request, but it runs on a server, reads at machine speed and often leaves from a cloud IP. For teams building AI agents that use the web, the result is a failure nobody planned for: the agent reaches a wall, and what happens next depends on details most tool layers ignore.
This guide covers why agents fall between the categories, what the wall looks like from inside a tool call, and how to build an agent that handles it honestly.
Neither crawler nor person
Vendors now describe these agents as a separate class. OpenAI says ChatGPT-User “is not used for crawling the web in an automatic fashion”. Google says Google-Agent is used “to navigate the web and perform actions upon user request”. Anthropic has a Claude-User agent for when people ask Claude questions.
The rules have not caught up, and the vendors do not agree on them:
| Vendor | User-triggered fetches and robots.txt |
|---|---|
| OpenAI | ”robots.txt rules may not apply” |
| ”generally ignore robots.txt rules” | |
| Anthropic | Its bots honour robots.txt directives |
RFC 9309 adds that robots.txt rules “are not a form of access authorization”, so they were never going to settle this alone. Meanwhile the traffic is growing: Cloudflare reports that AI “user action” crawling grew more than 15 times in 2025, and Thales’ 2026 Bad Bot Report puts bots at over 53% of all traffic in 2025. Sites have more reason than ever to filter automation, and agents get caught in that filter.
What the wall looks like from a tool call
| What comes back | What it means | What the agent should do |
|---|---|---|
| 200 with a challenge page | A bot check, not the page | Report that it could not see the page |
| 401 or a login page | The content needs an account | Ask the user to sign in |
| 402 | The site wants payment | Ask before paying anything |
| 403 | The server refuses | Report it, and do not hammer the page |
| 429 | Too many requests | Honour Retry-After, or stop |
| 404 or 410 | The page does not exist | Do not retry |
The first row is the dangerous one. A 403 at least looks like a failure. A challenge page often arrives as a 200 with a well-formed HTML body, so a simple fetch tool hands it to the model as if it were the article.
That matters because of how models behave with bad tool output. A September 2026 preprint on tool-augmented agents found that when a tool returned status:error, dishonesty was absent (0.0%), but when it returned status:ok with a redacted, corrupted, stale, malformed, empty or truncated value, it reached 45.3%. That study tested internal systems rather than web pages, so applying it to block pages is our inference. Still, the lesson carries over: an honest error is far safer than a success that holds nothing useful. We have also covered how challenge pages taint what AI tools produce.
A fetch tool that tells the truth
The fix belongs in the tool layer. The function below classifies each response before the model sees it, and returns page text only when the page is real. It needs only requests. We ran it on Python 3.9 against a local test server that returns each case.
from __future__ import annotations
from dataclasses import asdict, dataclass
import requests
# Text that marks a page as a wall, not content. Extend it from your own logs.
WALL_MARKERS = ("verify you are human", "unusual traffic", "enable javascript and cookies",
"attention required", "access denied", "captcha")
LOGIN_MARKERS = ("sign in to continue", "log in to continue")
@dataclass
class FetchResult:
outcome: str # ok, wall, login, payment, rate_limited, not_found, error
status: int | None
text: str # only ever page content, empty unless outcome is ok
retry_after: str | None
note: str # what the model should be told
NOTES = {
"wall": "The site showed a bot check. You did not see this page. Say so, do not guess its content.",
"login": "This page needs a login. Ask the user to sign in or to provide the content.",
"payment": "The site asks for payment (HTTP 402). Ask the user before paying anything.",
"rate_limited": "The site asked you to slow down. Wait, or tell the user it is unavailable now.",
"not_found": "The page does not exist. Do not retry it.",
"error": "The fetch failed. You did not see this page. Say so, do not guess its content.",
"ok": "",
}
def classify(status, body):
low = body.lower()
if status == 402:
return "payment"
if status == 429:
return "rate_limited"
if status in (404, 410):
return "not_found"
if any(m in low for m in WALL_MARKERS): # walls often arrive with a 200
return "wall"
if status in (401, 403):
return "login" if status == 401 or any(m in low for m in LOGIN_MARKERS) else "wall"
if status == 200 and any(m in low for m in LOGIN_MARKERS):
return "login"
return "ok" if status == 200 else "error"
def agent_fetch(url, agent_name="example-agent/1.0 (+https://example.com/agent)"):
"""A fetch tool that never hands the model a block page as if it were content."""
try:
r = requests.get(url, headers={"User-Agent": agent_name}, timeout=(10, 30))
except requests.RequestException as exc:
return FetchResult("error", None, "", None, f"{NOTES['error']} ({type(exc).__name__})")
outcome = classify(r.status_code, r.text)
return FetchResult(outcome, r.status_code, r.text if outcome == "ok" else "",
r.headers.get("Retry-After"), NOTES[outcome])Output, as status, outcome, whether the model receives page text, and Retry-After:
/article 200 ok content
/challenge 200 wall -
/blocked 403 wall -
/members 200 login -
/priced 402 payment -
/busy 429 rate_limited - 120
/gone 404 not_found -
/broken 503 error -
(no server)None error -Only one of the nine cases gives the model any page text. The /challenge row is the one a plain fetch gets wrong, since it returned a 200. Treat the marker lists as a starting point and grow them from your own logs, because no list catches every wall. If you serve tools through an MCP server, the specification has a place for this: execution errors are reported with isError: true, and clients “SHOULD provide tool execution errors to language models”.
Do not solve the CAPTCHA
Vendors are unusually aligned on this one. OpenAI described Operator as trained to ask the user to take over for logins, payment details and CAPTCHAs. Anthropic’s Claude in Chrome docs say that at a login page or CAPTCHA it “pauses and asks you to handle it manually”. Google’s recommended computer use instructions say “You MUST NEVER attempt to solve or bypass” a list that includes “CAPTCHAs (of any kind)”.
A CAPTCHA is a site asking for a human, so the right response is to hand over to one. That also makes for a better product: the person knows what happened and can decide.
Identity is the way through
The longer-term answer is for agents to say who they are in a way sites can verify. Web Bot Auth lets a client cryptographically sign its requests with HTTP Message Signatures. OpenAI already signs requests from its cloud browser, with a Signature-Agent header set to https://chatgpt.com, and Google says it is experimenting with the protocol. Cloudflare introduced a signed agents category in August 2025, and since July 2026 tracks the difference from verified bots as direct versus intermediary access. The IETF work is still a draft, so expect changes.
If you are building an agent today, send a descriptive User-Agent with contact details, publish what your agent does, and plan for signing.
The legal picture is unsettled
The first big case shows how open this is. Amazon sued Perplexity in November 2025 over its agentic browser, alleging violations of the US Computer Fraud and Abuse Act and its California equivalent, and the opinion says the core of the dispute was Perplexity’s decision not to use a user-agent string. In August 2026 the Ninth Circuit vacated a preliminary injunction, finding Amazon unlikely to show that Perplexity “accessed” its computers under that Act.
The court was careful about how far that goes: “We do not establish a new legal regime governing agentic AI.” It also said the outcome does not impair Amazon’s ability to regulate access through its terms of service. So this was not a licence for agents to ignore a site’s wishes. This is general information, not legal advice, and the case has gone back to the district court.
Building for the wall
- Assume walls are normal. Design the agent’s plan around pages it cannot reach, with fallbacks such as another source or asking the user.
- Classify before the model sees anything. Use a tool like the one above, and log the outcomes so you can see which sites fail.
- Offer official routes first. Where a site has an API or data product, use it. It is more reliable than a browser session.
- Keep sessions coherent. Multi-step tasks break when the IP changes midway. Our guide to proxies for AI agents that browse the web covers rotation and sticky sessions.
- Know what a proxy is for. Residential proxies help an agent see public pages as local visitors do, in the right country and from a consistent address. They do not change what a site allows, and they are not a way past a site that has said no. If a site wants a login, a payment or a person, respect that.
The bottom line
Agents reached the web before the web had a category for them, and for now the wall is part of the job. Treat it as a normal outcome: classify it in the tool layer, tell the model and the user the truth, hand CAPTCHAs back to people, and identify your agent. An agent that says “I could not see that page” is worth far more than one that guesses.
Sources and references
- OpenAI, bots documentation: developers.openai.com
- OpenAI, ChatGPT Work’s cloud browser allowlisting: help.openai.com
- OpenAI, Introducing Operator (23 January 2025): openai.com
- Google, user-triggered fetchers: developers.google.com
- Google, Gemini computer use: ai.google.dev
- Anthropic, on crawling and site owners: support.claude.com
- Anthropic, Claude in Chrome: code.claude.com
- Cloudflare, signed agents (28 August 2025, docs updated November 2026): blog.cloudflare.com, developers.cloudflare.com
- Cloudflare Radar 2025 Year in Review: blog.cloudflare.com
- Thales, Bad Bot Report 2026: cpl.thalesgroup.com
- IETF, Web Bot Auth protocol draft: datatracker.ietf.org
- RFC 9309, Robots Exclusion Protocol: rfc-editor.org
- RFC 9110 and RFC 6585, HTTP status codes: rfc-editor.org, rfc-editor.org
- Model Context Protocol, tools specification: modelcontextprotocol.io
- Sethi et al., Fabrication After Tool Failure (preprint, September 2026): arxiv.org/abs/2609.14758
- Ninth Circuit, Amazon.com Services v. Perplexity AI, No. 26-1444 (4 August 2026): uscourts.gov