llms.txt is the most talked-about file in AI search, and one of the least measured. It is a Markdown file at the root of a site that tells AI tools what the site is and where its most useful pages are. Plenty of people recommend it; Google says you can ignore it. For teams that monitor search visibility, the useful question is simpler: who actually has one?
On 16 November 2026 we checked the top 10,000 domains in the Tranco ranking, the same list we used for our study of who blocks AI crawlers. Here is what we found.
The headline numbers
| Domains | Share | |
|---|---|---|
| Top 10,000 domains checked | 10,000 | |
| Running a website (answered our request) | 7,865 | |
| With a real llms.txt | 929 | 9.3% of all, 11.8% of websites |
| Also with an llms-full.txt | 150 | 16% of those with llms.txt |
A vendor scan of the same kind of list in May 2026 found 586 valid files, 5.86% of the top 10,000. Methods differ slightly, but the direction is clear: adoption rose by roughly half in six months.
Half of the “200 OK” responses are not llms.txt files
2,068 domains answered our request for /llms.txt with a 200 OK status. Fewer than half of them, 929, served an actual llms.txt. The rest returned their home page, an error page dressed as a success, or a redirect to somewhere else. We also found 128 plain text files with no title, which the specification requires.
So counting status codes overstates adoption by more than double. We counted a file as real only if it was text, not HTML, stayed at an llms.txt address after redirects, and began with the Markdown title the specification calls “the only required section”.
Bigger sites, and developer sites, lead
| Rank | Websites | With llms.txt |
|---|---|---|
| Top 100 | 73 | 12.3% |
| Top 1,000 | 733 | 15.8% |
| 1,001 to 5,000 | 3,048 | 12.4% |
| 5,001 to 10,000 | 4,084 | 10.7% |
Among the earliest adopters in the ranking are Cloudflare, GitHub, Adobe, Samsung, Dropbox, PayPal, Shopify, Stripe, Cisco and Slack. The pattern is developer platforms and software companies: sites whose documentation people already paste into AI coding tools.
Domain endings tell the same story. 25.7% of .io websites have an llms.txt, and 15.4% of .com sites, against 2.5% of .edu, 1.8% of .jp and 1.0% of .gov. Docs platforms explain part of it: Mintlify has generated llms.txt for the docs sites it hosts since late 2024, and Yoast SEO added it for WordPress in June 2025.
The surprise: sites with llms.txt rarely block AI crawlers
We joined the results with the robots.txt files from our October study. The two groups look very different.
| AI crawler blocked in robots.txt | Sites with llms.txt | Sites without |
|---|---|---|
| GPTBot (OpenAI) | 4.9% | 16.9% |
| ClaudeBot (Anthropic) | 4.2% | 15.7% |
| Google-Extended | 3.0% | 13.8% |
| CCBot (Common Crawl) | 6.5% | 18.0% |
| At least one training crawler | 7.4% | 20.7% |
Sites that publish an llms.txt are about three times less likely to block AI training crawlers. That makes sense: an llms.txt is an invitation, and the companies sending it mostly want AI tools to understand their products.
A few sites send mixed signals. Twelve publish an llms.txt while blocking GPTBot, ClaudeBot and PerplexityBot in robots.txt, among them Bloomberg and the Daily Mail. The robots.txt data is from early October, so some may have changed since, but the point stands: an llms.txt lists content, it does not grant permission. Our guide to robots.txt and AI opt-outs covers the files that do.
Does anyone read it?
Here the picture is weaker than the adoption curve suggests.
- Google Search does not use it. Google’s guide to AI features says you “can ignore tactics like ‘chunking’ content, creating unnecessary AI text files (like llms.txt)”, while noting that it is “completely fine” to create them for other services that use them. Earlier, John Mueller compared llms.txt to the keywords meta tag.
- Chrome checks for it. Lighthouse now has an llms.txt audit, describing it as “an emerging convention used to provide a machine-readable summary of a website’s content”. A missing file is marked not applicable, “as providing the file is optional at the moment”.
- Crawlers rarely fetch it. No major AI company documents its crawlers reading other sites’ llms.txt files. In Ahrefs’ log study, 97% of about 38,000 files received no requests at all in May 2026.
The main readers today are AI coding assistants and agents pointed at documentation, where a clean list of pages is genuinely useful.
What to do with this
- Add one if you have documentation or a product with many pages. It is cheap, and agents and coding tools do use it. Keep it short: a title, a one-line summary, and sections linking to your most useful pages.
- Do not expect rankings. Nothing suggests llms.txt affects Google Search or AI Overviews. Our guide to AI Mode and rank tracking covers what does move AI visibility, and residential proxies let you check how AI answers describe you in each market.
- Make it real. Serve plain text at
/llms.txt, not your home page, and check after every CMS update. More than half the “200 OK” responses we saw were not llms.txt files. - Keep the signals consistent. If your llms.txt invites AI tools to read your pages, check that robots.txt does not shut out the same tools, or the other way round.
- Measure, then decide. Watch your server logs for requests to
/llms.txt. If nobody fetches it after a few months, you have your answer.
How we did it
On 16 November 2026 we requested https://<domain>/llms.txt, then the www. address if the first failed, for each of the top 10,000 domains in the Tranco ranking (list Q2K34). We used one request per address with a descriptive user agent, followed redirects, and read at most 2 MB. A domain counted as running a website if it returned any HTTP response. A file counted as a real llms.txt if it was not HTML, stayed at an llms.txt address and began with a Markdown title. The robots.txt comparison uses our scan of the same domains from 6 October 2026.
The bottom line
About one in eleven of the top 10,000 websites now has a real llms.txt, up roughly half since May, led by developer platforms and the biggest sites. Those sites are three times less likely to block AI crawlers. But Google Search ignores the file and crawlers rarely fetch it, so treat llms.txt as a cheap convenience for AI tools and agents, not as an SEO lever.
Sources and references
- llms.txt proposal and specification: llmstxt.org
- Google Search Central, Optimizing for AI features in Search: developers.google.com
- Chrome Lighthouse, llms.txt audit: developer.chrome.com
- Search Engine Journal, Google says llms.txt comparable to keywords meta tag (17 April 2025): searchenginejournal.com
- Ahrefs, llms.txt log study: ahrefs.com
- Thunderbit, llms.txt adoption in the Tranco top 10,000 (May 2026): thunderbit.com
- Common Crawl, content analysis of llms.txt files (31 August 2026): commoncrawl.org
- Mintlify, llms.txt support (20 November 2024): mintlify.com
- Yoast SEO 25.3 release (10 June 2025): yoast.com
- Tranco, list Q2K34: tranco-list.eu