Short answer
AI crawlers almost never request llms.txt. In a month of server logs from 839 sites, AI bots that passed an IP check made 4,301,434 requests, and exactly one of them asked for an llms file: GPTBot, on 20 October 2026, on a site that really has the file. The same bots requested robots.txt 278,704 times and sitemaps 355,413 times.
Requests for llms.txt do happen, 19,031 of them across 781 sites. But 57% come from clients that call themselves SEO audit tools, and another 34% from browser-like clients. And the response often lies: on 544 of my 838 sites, /llms.txt ends on the home page with a 200, which a quick check reads as "the file exists".
The rest of this article covers what the file is supposed to do, how I counted, who actually asks for it, the 200 trap, and how to check your own site with the files below.
What llms.txt is and what the proposal promises
llms.txt was proposed by Jeremy Howard on llmstxt.org, published on 3 September 2024. I read the current version, v2, modified on 10 August 2026, in full on 8 November 2026. It describes a markdown file at the root of a site that tells a language model where the site's key content is.
The format is short. Only the first element is required:
# Site name
> One-paragraph summary of the site (optional)
## Docs
- [Getting started](https://example.com/start): what the page covers
- [API reference](https://example.com/api): endpoints and limits
## Optional
- [Changelog](https://example.com/changelog)
Four statements from the proposal matter for SEO:
- The file is meant to be read on demand, when an agent needs information to help a user.
- The proposal expects it to be mainly useful at inference time, when a model answers, rather than for training.
- The text does not promise rankings. The word "rank" does not appear on the page.
/llms-full.txt, one long file with the full text, is a common convention, but the llmstxt.org text does not define it.
Publishing the file is easy. According to llmstxt.org, Yoast, All in One SEO and Wix generate it; I did not check each vendor separately. OpenAI, Anthropic and Google publish their own for developer documentation, and all three answered 200 with a file starting with # when I opened them on 8 November 2026. Chrome's Lighthouse has an audit for llms.txt: it flags a server error, while a 404 is reported as "not applicable", and the documentation calls the file optional for now.
So the file exists in the wild, from WordPress plugins to the developer docs of three AI labs. What a site owner needs to know is who actually requests it.
How I counted: 839 sites, IPs checked against official lists
I run 839 sites on four servers, in several countries and languages, mostly small and mostly in Europe and nearby. This is not a study of large sites, and I come back to that limit at the end.
| Parameter | Value |
|---|---|
| Window | 9 October to 8 November 2026, 31 days (29.4 days of logs per site on average) |
| Requests in the logs | 113,780,902 |
| Bots verified by IP | Google, Bing, Apple, OpenAI, Perplexity, Anthropic: 2,585 published IP ranges, snapshot of 8 November 2026 |
| Requests that claim an AI bot name from an IP outside its list | 1,606,546, counted as a separate pile |
| Sites behind Cloudflare | 784 of 838; on 774 a test request came back cf-cache-status: DYNAMIC, so it reached the server |
Three rules kept the numbers honest.
- A user agent is only text. I trust a bot's name only when its IP falls into the operator's published ranges. A "GPTBot" from an unknown address goes into the separate pile.
- My own checks are removed. On 8 November my checker requested
/llms.txtand/llms-full.txton every one of my sites and landed in the last day of the window. I recounted from the server logs and removed its 1,536 requests. - Everything is counted twice. The first count used my classifier, the second plain
zcatandgrepover the same logs. Both found the same single request from an AI bot to an llms file.
If you want the general method for reading AI crawler traffic in your own logs, I describe it in the article on log file analysis for AI bots and Googlebot.
Verified AI bots: robots.txt and sitemaps yes, llms.txt once
AI bots are active on these sites. Verified requests reached 783 of 839 sites:
| Bot (IP verified) | Requests | What it is |
|---|---|---|
| ClaudeBot | 2,282,554 | Anthropic crawler |
| GPTBot | 1,150,298 | OpenAI crawler |
| OAI-SearchBot | 497,797 | OpenAI search crawler |
| PerplexityBot | 344,698 | Perplexity crawler |
| ChatGPT-User | 18,105 (197 sites) | fetches a page when a person asks ChatGPT |
| Claude-SearchBot | 6,652 | Anthropic search crawler |
| Claude-User | 1,330 (476 sites) | fetches a page when a person asks Claude |
| Total | 4,301,434 |
To see whether these 4.3 million requests include service files at all, I used robots.txt and sitemaps as a control:
| What the same bots requested | Requests |
|---|---|
| robots.txt | 278,704 |
| sitemap files | 355,413 |
| other URLs | 3,667,316 |
| llms.txt, llms-full.txt, /.well-known/llms.txt | 1 |
The bots clearly go looking for the files that steer a crawl. They do not go looking for llms.txt. The single exception is GPTBot on 20 October 2026, answered with a 200 on one of the two sites where the file really exists. On those two sites verified AI bots made 4,597 requests in the month, and one of them was for the file.
The pile of requests that only claim an AI bot name, 1.6 million of them, asked for llms files zero times. Anyone faking a bot name here was not interested in the file either.
An independent Ahrefs study, published on 15 June 2026, covered 137,210 domains in Ahrefs Web Analytics for May 2026: 97% of about 38,000 valid llms.txt files got no requests at all that month, and the authors note that their customers are more technical and SEO-aware than the web at large. My sample is different, 839 of my own sites and their server logs, and the conclusion is the same: AI crawlers almost never fetch the file.
Who actually requests llms.txt
Across all sites there were 19,031 requests for llms files after removing my own checker, from about 3,250 IP addresses, on 781 sites:
| Path | Requests |
|---|---|
/llms.txt | 17,953 |
/.well-known/llms.txt | 584 |
/llms-full.txt | 493 |
/en/llms.txt | 1 |
So nearly every site gets asked. Here is who was asking:
| Class of client | Requests | Share |
|---|---|---|
| user agent names an SEO audit tool | 10,909 | 57.3% |
| ordinary browser string | 6,432 | 33.8% |
| other bots (bot, crawler, spider, scan in the name) | 1,514 | 8.0% |
| everything else | 175 | 0.9% |
| AI bot verified by IP | 1 | under 0.1% |
One SEO tool alone sent 8,526 of the 10,909 SEO-tool requests, 78%. The class is assigned by user agent, not by IP, so these clients "name themselves" SEO tools. My reading is that llms.txt has become a line on an audit checklist. Some audit tools do list a missing llms.txt as a problem in their reports. My clients do not see that line, because the audit tool I use for them does not have it.
The 6,432 browser-like requests I cannot split into people and scripts. A developer's agent working from a laptop also sends an ordinary-looking request from that laptop, so part of this class can be exactly the use the proposal describes.
What did they get back? 13,085 requests (68.8%) were redirected with a 301, 5,422 (28.5%) got a 404, 211 a 307, 42 a 302, and only 207 (1.1%) a 200. Even those 207 are mostly not the file: the two sites that have one received 72 requests for llms files of any status, so at least 135 of the 207 responses with a 200 came from sites without a file.
The 200 trap: a page that looks like a file
The usual quick test for "does this site have llms.txt" is the status code. That test is wrong for a large share of sites, because many servers answer unknown addresses with a page and a 200, or redirect them to the home page first.
On 8 November 2026 I requested /llms.txt on all 838 of my own sites, with up to three redirects:
| Result | Sites |
|---|---|
| real file | 2 |
| HTML page with status 200 (after a 301 to the home page) | 544 |
| 404 | 245 |
| error or no answer | 47 |
A status-only check would report 546 sites with a file. There are 2.
The same check on 28 large public sites on the same day, one request per file with the user agent ADLAB-research (+https://adlab.ee):
| Class | /llms.txt | Sites |
|---|---|---|
| real | 3 | Bloomberg, GitHub, Shopify |
| html200 | 2 | Reddit answers 200 with a web page; Forbes redirects to a consent page |
| 404 | 17 | news, retail and reference sites |
| blocked or error | 6 | 401, 403, 202 or a timeout: not measurable from this client |
Of the 22 sites I could measure, 3 serve a real file, 14%. /llms-full.txt is real on GitHub and Shopify. All three real files start with an H1 and a blockquote summary. Bloomberg's file was 48,790 bytes with 27 H2 sections and 141 links, GitHub's 28,658 bytes with 117 links, Shopify's 12,801 bytes with 72 links.
Three checks of the same 28 sites on 8 and 9 November gave different answers, and that is a lesson on its own:
- My first pass found two real files, not three: Bloomberg did not answer it. A second, independent implementation found all three.
- A curl cross-check right after the script matched 51 of 56 answers. Bloomberg sent the file at 21:47 UTC and a 403 about four minutes later. One large news site answered a 200 page over HTTP/2 and a 404 over HTTP/1.1.
- A repeat on 9 November matched 53 of 56 answers; Bloomberg answered 403 all day.
So treat a single check as a data point with a timestamp, not a verdict. The same applies to robots.txt, where a 200 with an HTML page also breaks the rules silently; I covered that case in the article on robots.txt returning a redirect, 404 or 5xx.
How to check your own llms.txt in two minutes
The llms.txt checker is one Python file for Python 3.8 or newer, with no packages to install. It requests /llms.txt and /llms-full.txt, reads the body and sorts the answer into a class:
| Class | Rule |
|---|---|
real | 200, the body is not HTML, the text starts with # |
html200 | 200, but the body is a web page: soft 404, consent page, home page |
404 | 404 or 410 |
blocked | 401, 403, 202, 429 or a redirect to a block page: not measurable |
error | no HTTP answer: DNS, TLS, timeout, more than 3 redirects |
other | a 200 that is not HTML and not markdown, an empty body, a 5xx |
Run it like this:
python3 check_llms_txt.py --selftest
python3 check_llms_txt.py example.com
python3 check_llms_txt.py sites.txt --out result.csv
python3 check_llms_txt.py --file llms.txt
The last line checks a draft before it goes live. For a real file the script also reports whether the first line is an H1, whether there is a blockquote, and how many H2 sections and links the file has. On one domain the run takes about 5 seconds; the list of 28 large sites takes about 4 minutes because of a 2-second pause between requests. The dataset of 28 sites is the CSV of the run described above, with the class, status, content type and size of each answer.
The second half of the check is your logs. The llms.txt log report is an SQL add-on to my server log audit kit. It splits requests for the three file locations into seven kinds of client: AI bot with a confirmed IP, a bot name from a foreign IP, AI bot without a published list (Meta, Amazon, ByteDance, Common Crawl), search engine bot, SEO tool, browser, script. Next to it sits the control: the same clients on robots.txt and sitemaps. On the synthetic sample log of 1,127 lines the report runs in 0.2 seconds after the kit.
If you only want a quick count without the kit, this line for nginx or Apache combined logs counts requests by status and client name and never prints visitor IPs:
grep -hE '"GET /(llms\.txt|llms-full\.txt|\.well-known/llms\.txt)(\?[^ ]*)? ' /var/log/nginx/access.log \
| awk -F'"' '{split($3,a," "); print a[1], substr($6,1,50)}' | sort | uniq -c | sort -rn | head -20
Two things to watch. Pass your own server and monitoring IPs to the kit, or your checks count as clients, as mine did on 8 November. And if a CDN answers /llms.txt from cache (cf-cache-status: HIT on Cloudflare), those requests never reach your log.
The log half does not work for sites that write no access logs at all, such as sites on Vercel or Cloudflare Workers, and for log formats other than nginx or Apache combined without an extra parser in the kit.
How to build llms.txt from a sitemap with an AI agent
If you decide to publish a file, the prompt for Claude Code or Codex walks an agent with a terminal through seven steps: five questions to you, a check of what /llms.txt answers now, a draft from your sitemap, a format check, your review, your upload, and a log command for the next 30 days.
The rules that make the draft trustworthy:
- Nothing invented. Every title and description is copied from the page itself: title, meta description or H1. A page without a meta description gets a bare link and goes onto a "needs text from me" list.
- Only indexable pages. Pages with
noindex, redirects, a canonical pointing elsewhere or a Disallow in robots.txt are skipped, with the reason written topages.tsv. - The agent publishes nothing. It does not upload, commit or deploy. You put the file at the web root yourself, served as
text/plainortext/markdown, and the agent runs the checker again. - No promises. The prompt forbids the agent from claiming that the file improves rankings or AI citations.
I tested the steps by hand on a local sandbox site of 5 pages: 4 went into the draft, the page with noindex was skipped, and one link stayed bare because its page had no meta description. After upload the checker reported real, 200, text/plain, 536 bytes. A full run by an agent has not been done, so watch the first steps and stop it if it writes outside its work folder. This kind of agent workflow is what I build for clients under AI automation.
Should you write llms.txt: my position
This part is my opinion, not a finding from the data.
I do not write these files for my own sites, and when clients ask for one, I talk them out of it. In the check above, 836 of my 838 sites do not serve one, and I see no reason to change that. For an ordinary business site it is work with no measurable payoff: neither the proposal nor my logs give a reason to expect crawler visits or rankings from it. The logs do not prove me right either, they only show that the crawlers do not ask.
The one exception I make is developer documentation or a public API. That is the case the proposal describes, coding agents read documentation on demand, and the file costs little to generate.
Whatever you decide, do not leave a fake. The address should answer either a real file or a plain 404, not your home page with a 200. On nginx a file-or-404 rule for this one address looks like this:
location = /llms.txt { try_files $uri =404; }
On a CMS, look for the redirect rule that sends unknown URLs to the home page and exclude the address from it. A missing file costs nothing; a fake one misleads every checker and audit tool that looks at it. If you are thinking about AI search more broadly, the files you publish matter less than the content structure I describe in the articles on generative engine optimization and AEO vs GEO, and the basics of sitemap.xml and robots.txt are still what AI crawlers actually request.
What this data cannot tell you
- The sample is mine. 839 mostly small sites, mostly in Europe and nearby. Large sites with heavy traffic could look different.
- Only two of my sites have a real file. Everything about "even when the file exists" rests on those two.
- Only bots with published IP ranges are verified. Meta, Amazon and other operators without public lists are not in the 4.3 million.
- Tools on a developer's machine are invisible as such. They send requests from that machine and look like a browser or a script in the logs, and the proposal names documentation for coding agents as the main use.
- A request is not use, and no request is not proof of no use. I cannot see inside OpenAI, Anthropic or Google. I see what reaches my servers, and I make no claim about whether llms.txt helps or hurts citations.
What the data does say: on these sites, in this month, 4.3 million verified AI bot requests skipped llms.txt with a single exception, and most of the traffic to the file came from audit tools. If you want to see the same picture for your own site, with your logs, crawl data and the soft 404s that hide behind a 200, I include it in a technical SEO audit.
