AI features are now part of most products: a chat assistant on the help page, AI search inside a store, generated product descriptions, a recommendation panel. Each of them can answer differently depending on where the user is, and those differences rarely show up when the team tests from the office. If your product serves several markets, checking AI output per country belongs next to your AI and machine learning work and your release testing, not after a complaint from one market.
This guide covers why AI output changes by country, what to check, how to make test requests look like a local user, and how to compare answers that are never word-for-word the same.
Why AI output changes by country
Most of the differences come from the product around the model, not the model itself:
- Location as context. Many assistants add the user’s country or city to the prompt, read from the IP. A wrong lookup gives a confident answer for the wrong market.
- Regional routing. Requests may go to a different model, version or endpoint per region, often for data residency reasons. Two markets can run different model versions on the same day.
- Feature flags. AI features are often launched country by country. A button that works in one market can be missing or broken in another.
- Policy and compliance rules. Local rules on health, finance, elections or age can trigger refusals or disclaimers in some countries only.
- Retrieval. When the assistant searches the web, a help centre or a product catalogue, it usually searches with the user’s region, so the facts and products it cites change.
The first and last of these depend on the IP the request comes from. That is why testing from one location misses them.
What to check in each market
For every market you serve, check the same set of properties:
| Check | What can go wrong |
|---|---|
| Language | Answers in English to a user in Japan, or mixed languages in one reply |
| Currency and units | Prices in dollars for a German user, miles instead of kilometres |
| Local facts | Wrong shipping times, holidays, tax rules or store locations for that country |
| Feature availability | The AI feature is hidden, errors out, or shows a “not available in your region” message |
| Policy behaviour | Refusals or disclaimers that should apply in one country appear everywhere, or the reverse |
| Products and sources | Recommendations or citations for items that are not sold or not relevant in that market |
| Formatting | Dates, numbers and addresses in the wrong local format |
Several of these overlap with classic page checks. Our guide to localisation QA at scale covers strings, hreflang and layout; this checklist covers what the AI layer adds on top.
Testing as a local user
Location usually comes from the IP, so the test request has to come from the market. Sending it through residential proxies targeted at the country makes the product apply the same region detection, flags and policies it applies to a real user there. Data center ranges are often mapped to the wrong country by IP databases, or put behind bot checks, which gives you test results nobody in that market would see.
Match the rest of the request to the market as well: the Accept-Language header, the timezone and the locale settings of the browser. A Japanese IP with an English browser is a different user from a Japanese IP with a Japanese browser, and both are worth testing. Our post on matching proxy geo, timezone and locale explains how to keep these signals consistent.
For a multi-turn chat, keep one IP for the whole conversation with a sticky session, so the product does not see the user move between turns:
customer-USERNAME-country-jp-sid-chat-ja-01-ttl-600:PASSWORDA small test harness
The harness below sends the same prompts to your AI endpoint from several markets and checks properties instead of exact text. Replace the endpoint and the checks with your own.
import requests
GATEWAY = "p.shifter.io:443"
USER, PASS = "USERNAME", "PASSWORD"
ENDPOINT = "https://app.example.com/api/assistant"
MARKETS = {
"de": {"lang": "de-DE", "currency": "€"},
"jp": {"lang": "ja-JP", "currency": "¥"},
"br": {"lang": "pt-BR", "currency": "R$"},
"us": {"lang": "en-US", "currency": "$"},
}
PROMPTS = ["How much is shipping for a 2 kg parcel?", "Recommend a laptop under my budget."]
RUNS = 3
def ask(country, lang, prompt, run):
proxy = f"http://customer-{USER}-country-{country}-sid-ai-{country}-{run}-ttl-600:{PASS}@{GATEWAY}"
r = requests.post(ENDPOINT, json={"message": prompt},
headers={"Accept-Language": lang},
proxies={"http": proxy, "https": proxy}, timeout=60)
return r.status_code, r.json().get("answer", "") if r.ok else ""
for country, m in MARKETS.items():
for prompt in PROMPTS:
hits = 0
for run in range(RUNS):
status, answer = ask(country, m["lang"], prompt, run)
if status == 200 and m["currency"] in answer:
hits += 1
print(country, prompt[:30], f"currency ok in {hits}/{RUNS} runs")Add a check per row of the table above: a language detector for the reply, a list of products sold in each market, a pattern for the regional “not available” message. Run each prompt several times, because model output varies between runs; a check that passes in two of three runs is a finding, not noise.
Making it part of release testing
- Run the harness after every model, prompt or flag change, from every market, not only from the default region.
- Keep a fixed prompt set per market, written by native speakers, including the questions customers really ask there.
- Store answers with country, model version and time, so you can tell a model change from a routing change.
- Review refusals by country each week. A refusal rate that jumps in one market usually points to a policy rule or a regional endpoint.
If your AI features also show up in search, the same per-market method applies to monitoring AI search visibility, where the question is what other assistants say about you in each country.
The bottom line
AI output is local, because the product around the model reads the user’s location and acts on it. Test each market from a residential IP in that market, with matching language and locale, keep multi-turn chats on one sticky IP, and check properties such as language, currency, local facts, availability and refusals across several runs. Do it on every release, and regional AI bugs stop being something your customers find first.