Discuss project

llms.txt for SEO: do ChatGPT and Claude bots request it?

In a month of logs from 839 sites, verified AI bots made 4.3 million requests and asked for llms.txt once. Who requests the file, and how to check yours.

Vladislav Krivorutško — founder of ADLAB
Vladislav Krivorutško

November 9, 2026 · 21 min read

From the ADLAB channel on YouTube. Watch on YouTube

Contents
  1. 01Short answer
  2. 02What llms.txt is and what the proposal promises
  3. 03How I counted: 839 sites, IPs checked against official lists
  4. 04Verified AI bots: robots.txt and sitemaps yes, llms.txt once
  5. 05Who actually requests llms.txt
  6. 06The 200 trap: a page that looks like a file
  7. 07How to check your own llms.txt in two minutes
  8. 08How to build llms.txt from a sitemap with an AI agent
  9. 09Should you write llms.txt: my position
  10. 10What this data cannot tell you
  11. Check yourself
  12. Files from the video

TL;DR - key points

  1. 1In server logs from 839 sites (9 October to 8 November 2026), AI bots verified by IP made 4,301,434 requests. One of them asked for llms.txt, while 278,704 asked for robots.txt and 355,413 for sitemaps
  2. 2All 19,031 requests for llms files came mostly from clients that name themselves SEO audit tools (57%) and from browser-like clients (34%); one tool alone sent 78% of the SEO-tool requests
  3. 3A 200 status does not prove the file exists: on 544 of my 838 sites the request for /llms.txt ends on the home page with a 200, and of 22 large sites I could measure, 3 serve a real file
  4. 4A real llms.txt is plain text whose first line is a markdown title. If you have no file, the address should answer a plain 404, not your home page
  5. 5The logs show who asked, not whether the file helps citations or rankings. I do not write these files for ordinary business sites; for developer docs and APIs it is a cheap exception

Short answer

AI crawlers almost never request llms.txt. In a month of server logs from 839 sites, AI bots that passed an IP check made 4,301,434 requests, and exactly one of them asked for an llms file: GPTBot, on 20 October 2026, on a site that really has the file. The same bots requested robots.txt 278,704 times and sitemaps 355,413 times.

Requests for llms.txt do happen, 19,031 of them across 781 sites. But 57% come from clients that call themselves SEO audit tools, and another 34% from browser-like clients. And the response often lies: on 544 of my 838 sites, /llms.txt ends on the home page with a 200, which a quick check reads as "the file exists".

The rest of this article covers what the file is supposed to do, how I counted, who actually asks for it, the 200 trap, and how to check your own site with the files below.

What llms.txt is and what the proposal promises

llms.txt was proposed by Jeremy Howard on llmstxt.org, published on 3 September 2024. I read the current version, v2, modified on 10 August 2026, in full on 8 November 2026. It describes a markdown file at the root of a site that tells a language model where the site's key content is.

The format is short. Only the first element is required:

# Site name

> One-paragraph summary of the site (optional)

## Docs

- [Getting started](https://example.com/start): what the page covers
- [API reference](https://example.com/api): endpoints and limits

## Optional

- [Changelog](https://example.com/changelog)

Four statements from the proposal matter for SEO:

  1. The file is meant to be read on demand, when an agent needs information to help a user.
  2. The proposal expects it to be mainly useful at inference time, when a model answers, rather than for training.
  3. The text does not promise rankings. The word "rank" does not appear on the page.
  4. /llms-full.txt, one long file with the full text, is a common convention, but the llmstxt.org text does not define it.

Publishing the file is easy. According to llmstxt.org, Yoast, All in One SEO and Wix generate it; I did not check each vendor separately. OpenAI, Anthropic and Google publish their own for developer documentation, and all three answered 200 with a file starting with # when I opened them on 8 November 2026. Chrome's Lighthouse has an audit for llms.txt: it flags a server error, while a 404 is reported as "not applicable", and the documentation calls the file optional for now.

So the file exists in the wild, from WordPress plugins to the developer docs of three AI labs. What a site owner needs to know is who actually requests it.

How I counted: 839 sites, IPs checked against official lists

I run 839 sites on four servers, in several countries and languages, mostly small and mostly in Europe and nearby. This is not a study of large sites, and I come back to that limit at the end.

ParameterValue
Window9 October to 8 November 2026, 31 days (29.4 days of logs per site on average)
Requests in the logs113,780,902
Bots verified by IPGoogle, Bing, Apple, OpenAI, Perplexity, Anthropic: 2,585 published IP ranges, snapshot of 8 November 2026
Requests that claim an AI bot name from an IP outside its list1,606,546, counted as a separate pile
Sites behind Cloudflare784 of 838; on 774 a test request came back cf-cache-status: DYNAMIC, so it reached the server

Three rules kept the numbers honest.

  1. A user agent is only text. I trust a bot's name only when its IP falls into the operator's published ranges. A "GPTBot" from an unknown address goes into the separate pile.
  2. My own checks are removed. On 8 November my checker requested /llms.txt and /llms-full.txt on every one of my sites and landed in the last day of the window. I recounted from the server logs and removed its 1,536 requests.
  3. Everything is counted twice. The first count used my classifier, the second plain zcat and grep over the same logs. Both found the same single request from an AI bot to an llms file.

If you want the general method for reading AI crawler traffic in your own logs, I describe it in the article on log file analysis for AI bots and Googlebot.

Verified AI bots: robots.txt and sitemaps yes, llms.txt once

AI bots are active on these sites. Verified requests reached 783 of 839 sites:

Bot (IP verified)RequestsWhat it is
ClaudeBot2,282,554Anthropic crawler
GPTBot1,150,298OpenAI crawler
OAI-SearchBot497,797OpenAI search crawler
PerplexityBot344,698Perplexity crawler
ChatGPT-User18,105 (197 sites)fetches a page when a person asks ChatGPT
Claude-SearchBot6,652Anthropic search crawler
Claude-User1,330 (476 sites)fetches a page when a person asks Claude
Total4,301,434

To see whether these 4.3 million requests include service files at all, I used robots.txt and sitemaps as a control:

What the same bots requestedRequests
robots.txt278,704
sitemap files355,413
other URLs3,667,316
llms.txt, llms-full.txt, /.well-known/llms.txt1
Verified AI bot requests: service files vs llms.txtall requests by these bots4,301,434sitemap files355,413robots.txt278,704llms files1GPTBot, 20 October 2026110010k1MLog scale: each gridline is ×10
Requests by AI bots whose IP matched the operator's published list, 839 sites, 9 October to 8 November 2026. The scale is logarithmic: on a linear one the llms.txt bar would not be visible. Data from the tables above.

The bots clearly go looking for the files that steer a crawl. They do not go looking for llms.txt. The single exception is GPTBot on 20 October 2026, answered with a 200 on one of the two sites where the file really exists. On those two sites verified AI bots made 4,597 requests in the month, and one of them was for the file.

The pile of requests that only claim an AI bot name, 1.6 million of them, asked for llms files zero times. Anyone faking a bot name here was not interested in the file either.

An independent Ahrefs study, published on 15 June 2026, covered 137,210 domains in Ahrefs Web Analytics for May 2026: 97% of about 38,000 valid llms.txt files got no requests at all that month, and the authors note that their customers are more technical and SEO-aware than the web at large. My sample is different, 839 of my own sites and their server logs, and the conclusion is the same: AI crawlers almost never fetch the file.

Who actually requests llms.txt

Across all sites there were 19,031 requests for llms files after removing my own checker, from about 3,250 IP addresses, on 781 sites:

PathRequests
/llms.txt17,953
/.well-known/llms.txt584
/llms-full.txt493
/en/llms.txt1

So nearly every site gets asked. Here is who was asking:

Class of clientRequestsShare
user agent names an SEO audit tool10,90957.3%
ordinary browser string6,43233.8%
other bots (bot, crawler, spider, scan in the name)1,5148.0%
everything else1750.9%
AI bot verified by IP1under 0.1%

One SEO tool alone sent 8,526 of the 10,909 SEO-tool requests, 78%. The class is assigned by user agent, not by IP, so these clients "name themselves" SEO tools. My reading is that llms.txt has become a line on an audit checklist. Some audit tools do list a missing llms.txt as a problem in their reports. My clients do not see that line, because the audit tool I use for them does not have it.

The 6,432 browser-like requests I cannot split into people and scripts. A developer's agent working from a laptop also sends an ordinary-looking request from that laptop, so part of this class can be exactly the use the proposal describes.

What did they get back? 13,085 requests (68.8%) were redirected with a 301, 5,422 (28.5%) got a 404, 211 a 307, 42 a 302, and only 207 (1.1%) a 200. Even those 207 are mostly not the file: the two sites that have one received 72 requests for llms files of any status, so at least 135 of the 207 responses with a 200 came from sites without a file.

The 200 trap: a page that looks like a file

The usual quick test for "does this site have llms.txt" is the status code. That test is wrong for a large share of sites, because many servers answer unknown addresses with a page and a 200, or redirect them to the home page first.

On 8 November 2026 I requested /llms.txt on all 838 of my own sites, with up to three redirects:

ResultSites
real file2
HTML page with status 200 (after a 301 to the home page)544
404245
error or no answer47

A status-only check would report 546 sites with a file. There are 2.

What /llms.txt returns on 838 sites/llms.txt on 838 of my sites544245544 sites301home page200 + HTMLstatus says file exists: soft 404245 sites404honest answer: no file2 sites200text “# …”real file47 siteserror or no answerStatus-only check: “has a file”546Real file2
Response to /llms.txt with up to three redirects followed, 838 of my sites, measured on 8 November 2026.

The same check on 28 large public sites on the same day, one request per file with the user agent ADLAB-research (+https://adlab.ee):

Class/llms.txtSites
real3Bloomberg, GitHub, Shopify
html2002Reddit answers 200 with a web page; Forbes redirects to a consent page
40417news, retail and reference sites
blocked or error6401, 403, 202 or a timeout: not measurable from this client

Of the 22 sites I could measure, 3 serve a real file, 14%. /llms-full.txt is real on GitHub and Shopify. All three real files start with an H1 and a blockquote summary. Bloomberg's file was 48,790 bytes with 27 H2 sections and 141 links, GitHub's 28,658 bytes with 117 links, Shopify's 12,801 bytes with 72 links.

Three checks of the same 28 sites on 8 and 9 November gave different answers, and that is a lesson on its own:

  • My first pass found two real files, not three: Bloomberg did not answer it. A second, independent implementation found all three.
  • A curl cross-check right after the script matched 51 of 56 answers. Bloomberg sent the file at 21:47 UTC and a 403 about four minutes later. One large news site answered a 200 page over HTTP/2 and a 404 over HTTP/1.1.
  • A repeat on 9 November matched 53 of 56 answers; Bloomberg answered 403 all day.

So treat a single check as a data point with a timestamp, not a verdict. The same applies to robots.txt, where a 200 with an HTML page also breaks the rules silently; I covered that case in the article on robots.txt returning a redirect, 404 or 5xx.

How to check your own llms.txt in two minutes

The llms.txt checker is one Python file for Python 3.8 or newer, with no packages to install. It requests /llms.txt and /llms-full.txt, reads the body and sorts the answer into a class:

ClassRule
real200, the body is not HTML, the text starts with #
html200200, but the body is a web page: soft 404, consent page, home page
404404 or 410
blocked401, 403, 202, 429 or a redirect to a block page: not measurable
errorno HTTP answer: DNS, TLS, timeout, more than 3 redirects
othera 200 that is not HTML and not markdown, an empty body, a 5xx

Run it like this:

python3 check_llms_txt.py --selftest
python3 check_llms_txt.py example.com
python3 check_llms_txt.py sites.txt --out result.csv
python3 check_llms_txt.py --file llms.txt

The last line checks a draft before it goes live. For a real file the script also reports whether the first line is an H1, whether there is a blockquote, and how many H2 sections and links the file has. On one domain the run takes about 5 seconds; the list of 28 large sites takes about 4 minutes because of a 2-second pause between requests. The dataset of 28 sites is the CSV of the run described above, with the class, status, content type and size of each answer.

The second half of the check is your logs. The llms.txt log report is an SQL add-on to my server log audit kit. It splits requests for the three file locations into seven kinds of client: AI bot with a confirmed IP, a bot name from a foreign IP, AI bot without a published list (Meta, Amazon, ByteDance, Common Crawl), search engine bot, SEO tool, browser, script. Next to it sits the control: the same clients on robots.txt and sitemaps. On the synthetic sample log of 1,127 lines the report runs in 0.2 seconds after the kit.

If you only want a quick count without the kit, this line for nginx or Apache combined logs counts requests by status and client name and never prints visitor IPs:

grep -hE '"GET /(llms\.txt|llms-full\.txt|\.well-known/llms\.txt)(\?[^ ]*)? ' /var/log/nginx/access.log \
  | awk -F'"' '{split($3,a," "); print a[1], substr($6,1,50)}' | sort | uniq -c | sort -rn | head -20

Two things to watch. Pass your own server and monitoring IPs to the kit, or your checks count as clients, as mine did on 8 November. And if a CDN answers /llms.txt from cache (cf-cache-status: HIT on Cloudflare), those requests never reach your log.

The log half does not work for sites that write no access logs at all, such as sites on Vercel or Cloudflare Workers, and for log formats other than nginx or Apache combined without an extra parser in the kit.

How to build llms.txt from a sitemap with an AI agent

If you decide to publish a file, the prompt for Claude Code or Codex walks an agent with a terminal through seven steps: five questions to you, a check of what /llms.txt answers now, a draft from your sitemap, a format check, your review, your upload, and a log command for the next 30 days.

The rules that make the draft trustworthy:

  • Nothing invented. Every title and description is copied from the page itself: title, meta description or H1. A page without a meta description gets a bare link and goes onto a "needs text from me" list.
  • Only indexable pages. Pages with noindex, redirects, a canonical pointing elsewhere or a Disallow in robots.txt are skipped, with the reason written to pages.tsv.
  • The agent publishes nothing. It does not upload, commit or deploy. You put the file at the web root yourself, served as text/plain or text/markdown, and the agent runs the checker again.
  • No promises. The prompt forbids the agent from claiming that the file improves rankings or AI citations.

I tested the steps by hand on a local sandbox site of 5 pages: 4 went into the draft, the page with noindex was skipped, and one link stayed bare because its page had no meta description. After upload the checker reported real, 200, text/plain, 536 bytes. A full run by an agent has not been done, so watch the first steps and stop it if it writes outside its work folder. This kind of agent workflow is what I build for clients under AI automation.

Should you write llms.txt: my position

This part is my opinion, not a finding from the data.

I do not write these files for my own sites, and when clients ask for one, I talk them out of it. In the check above, 836 of my 838 sites do not serve one, and I see no reason to change that. For an ordinary business site it is work with no measurable payoff: neither the proposal nor my logs give a reason to expect crawler visits or rankings from it. The logs do not prove me right either, they only show that the crawlers do not ask.

The one exception I make is developer documentation or a public API. That is the case the proposal describes, coding agents read documentation on demand, and the file costs little to generate.

Whatever you decide, do not leave a fake. The address should answer either a real file or a plain 404, not your home page with a 200. On nginx a file-or-404 rule for this one address looks like this:

location = /llms.txt { try_files $uri =404; }

On a CMS, look for the redirect rule that sends unknown URLs to the home page and exclude the address from it. A missing file costs nothing; a fake one misleads every checker and audit tool that looks at it. If you are thinking about AI search more broadly, the files you publish matter less than the content structure I describe in the articles on generative engine optimization and AEO vs GEO, and the basics of sitemap.xml and robots.txt are still what AI crawlers actually request.

What this data cannot tell you

  • The sample is mine. 839 mostly small sites, mostly in Europe and nearby. Large sites with heavy traffic could look different.
  • Only two of my sites have a real file. Everything about "even when the file exists" rests on those two.
  • Only bots with published IP ranges are verified. Meta, Amazon and other operators without public lists are not in the 4.3 million.
  • Tools on a developer's machine are invisible as such. They send requests from that machine and look like a browser or a script in the logs, and the proposal names documentation for coding agents as the main use.
  • A request is not use, and no request is not proof of no use. I cannot see inside OpenAI, Anthropic or Google. I see what reaches my servers, and I make no claim about whether llms.txt helps or hurts citations.

What the data does say: on these sites, in this month, 4.3 million verified AI bot requests skipped llms.txt with a single exception, and most of the traffic to the file came from audit tools. If you want to see the same picture for your own site, with your logs, crawl data and the soft 404s that hide behind a 200, I include it in a technical SEO audit.

Quiz

Check yourself

A few questions on the article. The answer and the reasoning appear as soon as you pick; nothing is sent anywhere.

0/4

answered

  1. 1.A checker shows that /llms.txt on your site answers 200. What does that tell you?
  2. 2.Your logs show hundreds of requests for /llms.txt in a month. Who most likely sent them?
  3. 3.An audit report flags a missing llms.txt on an ordinary business site. What does the author do?
  4. 4.A request for /llms.txt says GPTBot in the user agent but comes from an IP outside OpenAI's published ranges. How should you count it?

ADLAB · files from the video

Files from the video

Scripts, prompts, templates and data shown in the video. Text files open as plain text; the zip holds every file of one item.

  1. 01Script4 files · 24 KB · MIT

    llms.txt checker: a real file, or a 200 page that only looks like one?

    Python script, stdlib only: sorts /llms.txt and /llms-full.txt into real, html200, 404, blocked or error, and checks the llmstxt.org format.

    What's inside · 4 files
    • README.md

      3-step run, class rules, CSV columns, limits

      4.2 KB
    • check_llms_txt.py

      The checker: Python 3.8+, standard library only, with an offline self-test

      16 KB
    • curl_check.sh

      Cross-check: the same classes from curl and shell, separate code

      3.1 KB
    • sites.txt

      The 28 public domains of the ADLAB dataset, one per line

      607 B
  2. 02SQL · Script4 files · 253 KB · MIT

    llms.txt in your server logs: who requests it and what they get

    SQL add-on for log-audit-kit: counts requests for llms.txt by confirmed AI bot, fake bot name, SEO tool and browser, with robots.txt as a control.

    What's inside · 4 files
    • README.md

      What you need, the 3-step run, sections, limits

      9.0 KB
    • llms_report.sql

      8 sections over the kit's enriched.parquet: by kind of client, by bot, by day, with robots.txt as a control

      6.1 KB
    • sample/make_llms_sample.py

      Generator of the synthetic log (deterministic, stdlib only)

      10 KB
    • sample/llms.sample.log

      Synthetic log of 1,127 lines for example.com, 7 days, with llms requests from every kind of client

      228 KB
  3. 03Dataset4 files · 28 KB · CC-BY-4.0

    llms.txt on 28 large sites: who publishes a real file (data, 2026-11-08)

    28 large sites checked on 2026-11-08: 3 publish a real /llms.txt, 2 answer 200 with a web page, 17 return 404, 6 could not be measured. CSV, CC BY 4.0.

    What's inside · 4 files
    • llms-txt-28-sites.csv

      One row per site: class, status, content type, size and llmstxt.org format for /llms.txt and /llms-full.txt

      5.9 KB
    • check_llms_txt.py

      Python 3.8+, standard library only; reproduces the measurement (same file as in the checker resource)

      16 KB
    • sites.txt

      The 28 domains that were requested

      607 B
    • README.md

      Definitions, columns, the curl cross-check, limits

      6.0 KB
  4. 04Prompt2 files · 11 KB · MIT

    Prompt: have an AI agent check llms.txt and draft one from your sitemap

    A prompt for Claude Code or Codex: checks your /llms.txt, drafts one from your sitemap with text copied from your own pages, and publishes nothing.

    What's inside · 2 files
    • prompt.md

      The prompt, ready to paste into an agent with a terminal

      9.0 KB
    • README.md

      How to use it and what to check before you trust it

      2.0 KB

Frequently asked questions

What is llms.txt?
llms.txt is a markdown file at the root of a site, proposed by Jeremy Howard on llmstxt.org in September 2024. It lists a site's key pages with short notes so that a language model can find them when it answers a user. The only required element is an H1 title; a blockquote summary and H2 sections with link lists are optional. It is a proposal, not a standard adopted by search engines.
Does GPTBot or ClaudeBot read llms.txt?
In my logs from 839 sites over a month, AI bots whose IP matched the operators' published lists made 4.3 million requests and asked for llms.txt exactly once: GPTBot on 20 October 2026, on a site that has the file. ClaudeBot made 2.28 million requests and none for llms files. The same bots requested robots.txt 278,704 times. This describes my sample of mostly small sites, not every site on the web.
Does llms.txt help SEO or rankings in Google?
Nobody has shown that it does. The llmstxt.org text does not promise rankings, Chrome's Lighthouse documentation calls the file optional, and a missing file is marked 'not applicable' there. Server logs can show who requested the file, not whether it changed citations or rankings, so I do not expect any ranking effect from it.
Why does my llms.txt return 200 when I never created one?
Most often the server redirects unknown addresses to the home page or answers them with an ordinary page and status 200, a soft 404. On 544 of my 838 sites a request for /llms.txt ended on the home page with a 200. A checker that looks only at the status code counts that as a published file. The fix is a plain 404 for that address.
How do I check if my llms.txt is a real file?
Request /llms.txt with curl or a checker, not in a browser tab, and look at three things: the status, the content type and the first line. A real file answers 200, is not an HTML page and starts with a markdown title such as '# Site name'. The checker from the materials below does this for a list of domains in a few seconds per domain and also checks /llms-full.txt.
Who requests llms.txt if not AI bots?
In my sample, 10,909 of 19,031 requests for llms files came from clients whose user agent names an SEO audit tool, and 6,432 from clients with an ordinary browser string, which can be people or scripts. Developer tools that run on a laptop also look like a browser or a script in the logs, because they send requests from that laptop and not from published bot ranges.
Vladislav Krivorutško — founder of ADLAB

About the author

Vladislav Krivorutško

Founder of ADLAB OÜ · SEO and Google Ads

Over 20 years in search traffic and monetization, and on the Estonian market since 2017. I work solo: I run the audit, build the strategy and deliver the project myself — no subcontractors, no templates. I only write about what I have tested on my own and client sites.

  • 20+ years in search traffic
  • 50+ end-to-end projects
  • Own sites in competitive niches
  • SEO for ru/et/en in one market
More about me

Conclusion

llms.txt is a proposal for agents that answer users, and the AI crawlers on my 839 sites behave accordingly: hundreds of thousands of requests for robots.txt and sitemaps, one for llms.txt. The audit tools that flag a missing file request it far more often than any AI bot does. My rule is simple: no file and no false promises for an ordinary business site, a real file for developer documentation, and in every case either a real file or a plain 404 at that address. If your server answers with the home page there, or you want to know what AI bots really do on your site, that is part of a technical SEO audit.

Read next