Grounding is what turns a language model from something that sounds right into something that can show its work. A grounded answer is built from evidence retrieved at the time of the question, and it points to the sources it used, so a reader, or another system, can check it.
For agents that act on the web, grounding in live data is not optional. Their questions are about prices today, availability now, and what a page says this morning. This explainer covers what grounding is, how a live-web grounding loop works, and the places it breaks without anyone noticing. The case for why residential infrastructure matters to grounding is set out in why LLMs need residential proxies for AI grounding.
What LLM grounding is, and is not
| Approach | Where knowledge comes from | Freshness | Can cite sources |
|---|---|---|---|
| Model knowledge alone | Training data, fixed in the weights | As old as the training cut-off | No |
| Fine-tuning | New data trained into the weights | As old as the last training run | No |
| Retrieval over a private index | Documents indexed in advance | As fresh as the last indexing | Yes |
| Live web grounding | Pages searched and fetched at answer time | Current | Yes |
Grounding is the practice of constraining the answer to retrieved evidence and citing it. Retrieval over your own documents and live web grounding both qualify. What distinguishes live web grounding is that the evidence is fetched when the question is asked, which is what agents need for anything time-sensitive, local or outside their training data.
The grounding loop
A live-web grounding loop for an agent usually has seven steps.
- Plan the query. Turn the user’s request into one or more search queries, including the market and language the answer should reflect.
- Search. Get candidate sources from a search engine.
- Select. Choose which results to read, preferring primary and authoritative sources.
- Fetch. Retrieve the pages.
- Extract. Pull the relevant passages out of each page.
- Answer with citations. Generate the answer from the passages, citing each claim.
- Verify. Check that each claim is actually supported by the passage it cites.
Each step has a latency budget. Search and fetch dominate, and fetching several sources in parallel is usually what keeps an agent responsive.
Where grounding breaks
Most grounding failures do not produce errors. They produce confident answers built on bad evidence.
Blocked or challenged fetches. A fetch that returns a challenge page or an empty shell gives the model nothing to work with, and a model given nothing tends to fill the gap from memory. The fix is to make fetch failures explicit: pass the model a clear “this source could not be retrieved” rather than an empty context, and prefer answering with fewer sources over answering from invented ones.
Wrong market or language. A page fetched from the wrong country can show different prices, availability or even different products. If the agent is answering for a user in Madrid, the evidence should reflect Madrid. Matching language and timezone signals to the exit matters as well; see matching proxy geo, timezone and locale.
Stale caches. Caching fetched pages saves time and money, and a cache that outlives the freshness the question needs quietly returns yesterday’s facts. Set cache lifetimes by query type, short for prices and news, long for reference material.
Content that only exists after rendering. Many pages load their data with JavaScript. A plain fetch returns the frame without the facts; see when you need a web scraping API.
Instructions hidden in fetched pages. This is the security failure specific to agents. A web page can contain text written to look like instructions to the model. An agent that treats fetched content as instructions can be steered into leaking data or taking actions its user never asked for. Treat every fetched page as untrusted data, never as commands: keep retrieved text clearly separated from the agent’s instructions, restrict what tools the agent can call based on retrieved content, and require confirmation for consequential actions.
Citation drift. A cited page can change after the answer is given. Keep a snapshot or hash of the passage that was cited, so the citation remains checkable.
Where proxies and APIs fit in the loop
Two steps of the loop touch the open web: search and fetch.
Search is usually best handled by a SERP API, which returns structured results for a given location and device without the agent having to browse a search engine. The reasoning is in why AI agents need real-time SERP APIs.
Fetch is where proxy infrastructure matters. Agents fetch from many unrelated sites, in many markets, often in parallel, and the sites that matter most tend to be the ones that are defended. Residential exits in the user’s market return the page a local user would see. With the Shifter gateway, the market and session are set in the credentials against p.shifter.io:443:
customer-USERNAME-country-es-city-madrid:PASSWORD
customer-USERNAME-country-es-city-madrid-sid-task-5521-ttl-600:PASSWORDThe first line rotates the exit on every request, which suits fetching several independent sources in parallel. The second holds one exit for ten minutes, which suits a multi-page task on one site where consistency matters. The trade-off is in sticky vs rotating residential proxies.
For rendered pages, a web scraping API handles the browser, retries and extraction in one request, and charges only for successful responses. The product view of agent infrastructure is on the proxies for AI agents page, and the criteria for choosing a provider are in best proxies for AI agents that browse the web.
Freshness, cost and latency
Each grounded answer costs a number of searches plus a number of fetches, and each adds latency. Three practices keep both under control.
- Fetch in parallel and stop early once enough independent sources agree.
- Cache by question type, with lifetimes that match how quickly the underlying facts change.
- Prefer primary sources. One authoritative page is worth several pages that summarise it.
Evaluating a grounded agent
Measure grounding directly, not just answer quality.
| Metric | What it measures |
|---|---|
| Groundedness | Share of claims supported by the passage they cite |
| Citation accuracy | Whether the cited page actually contains the claim |
| Freshness | How recent the evidence was for time-sensitive questions |
| Fetch failure rate | How often evidence could not be retrieved, by site and market |
| Unsupported answer rate | How often the agent answered despite missing evidence |
Build a test set of time-sensitive and market-specific questions whose answers change, and rerun it on a schedule. A grounded agent that passes today can fail next month because a source changed its markup.
Staying on the right side of it
Agents browse on behalf of people, and should behave accordingly: respect site terms and crawler rules, keep request rates proportionate, do not bypass logins or paywalls, and keep a record of what was fetched and when. The wider framing is in ethical residential proxies for AI data collection.
FAQ
Is grounding the same as retrieval-augmented generation?
Retrieval-augmented generation is one way to ground. Grounding is the broader practice of tying answers to retrieved, citable evidence, whether from a private index or the live web.
Do agents need proxies to ground on live web data?
At low volume, perhaps not. At scale, across many sites and markets, fetches from a single address get throttled or blocked, and blocked fetches are the most common silent grounding failure.
How do we protect an agent from instructions hidden in web pages?
Treat fetched content as untrusted data, keep it separate from the agent’s instructions, limit which tools retrieved content can trigger, and require confirmation for consequential actions.
Should the agent browse search engines directly?
A SERP API is usually faster, cheaper and more reliable for the search step. Browsing is better reserved for fetching the pages the search returns.
The bottom line
Grounding makes an agent’s answers checkable by building them from evidence fetched at the time of the question and citing it. The loop is simple: plan, search, select, fetch, extract, answer, verify. The failures are quiet: blocked fetches filled in from memory, pages from the wrong market, stale caches, unrendered content, hidden instructions and citations that drift.
Make fetch failures explicit, fetch from the user’s market, cache by question type, treat retrieved text as untrusted, and measure groundedness directly. For how training data differs from grounding data, see how to build large-scale training datasets from the open web.