dataset

llms.txt on 28 large sites: who publishes a real file (data, 2026-11-08)

28 large sites checked on 2026-11-08: 3 publish a real /llms.txt, 2 answer 200 with a web page, 17 return 404, 6 could not be measured. CSV, CC BY 4.0.

v1.0
updated 8 November 2026
CC-BY-4.0
Vladislav Krivorutško

Download llms-txt-28-sites.csv Download all (.zip)Jump to the text

3 of 28
large sites publish a real /llms.txt
2
answer 200 with a web page instead
17
return 404
6
could not be measured: blocked or no answer

On 2026-11-08, 3 of 28 large sites published a real /llms.txt: bloomberg.com, github.com and shopify.com. Two more, forbes.com and reddit.com, answered with HTTP 200 and a web page, 17 returned 404, and 6 could not be measured because they blocked the request or did not answer. Among the 22 sites that could be measured, 3 publish the file, 14 %.

A check that looks only at the status code would have counted 5 sites, not 3. forbes.com and reddit.com send a 200 that is not a file. That gap is the reason for the class html200 in the llms.txt checker, the script behind this data.

Files in this resource

4 files · or all in one zip

llms-txt-28-sites.csv
One row per site: class, status, content type, size and llmstxt.org format for /llms.txt and /llms-full.txt
5.9 KB
check_llms_txt.py
Python 3.8+, standard library only; reproduces the measurement (same file as in the checker resource)
16 KB
sites.txt
The 28 domains that were requested
583 B
README.md
Definitions, columns, the curl cross-check, limits
5.9 KB

What was measured

The script requested /llms.txt and /llms-full.txt from 28 domains between 21:45:50 and 21:49:31 UTC on 2026-11-08, one GET per URL, with the User-Agent ADLAB-research (+https://adlab.ee), a 2 second pause between requests, gzip accepted and up to 3 redirects. The domains are the same 28 as in the robots.txt dataset.

An answer is real when it is HTTP 200, not an HTML page, and the text starts with #. It is html200 when the status is 200 but the body is a web page, 404 for 404 or 410, blocked for 401, 403, 202, 429 or a redirect to a block page, and error when there was no HTTP answer. blocked and error are results: the script does not try to get around them.

Results

Class	/llms.txt	/llms-full.txt	Sites
real	3	2	`/llms.txt`: bloomberg.com, github.com, shopify.com. `/llms-full.txt`: github.com, shopify.com
html200	2	2	forbes.com, reddit.com
404	17	17	17 sites for `/llms.txt` (nytimes.com, cnn.com, bbc.com, theguardian.com, techcrunch.com, wikipedia.org, amazon.com, ebay.com, walmart.com, stackoverflow.com, quora.com, nerdwallet.com, healthline.com, webmd.com, booking.com, cnet.com, zillow.com); bloomberg.com instead of techcrunch.com for `/llms-full.txt`
blocked	5	6	reuters.com (401), wsj.com, medium.com, tripadvisor.com (403), imdb.com (202); `/llms-full.txt` also techcrunch.com (403)
error	1	1	washingtonpost.com: both requests timed out after 20 seconds

The three real files

Site	/llms.txt size	H1 first	Blockquote	H2 sections	Links	/llms-full.txt
bloomberg.com	48,790 bytes	yes	yes	27	141	404
github.com	28,658 bytes	yes	yes	12	117	real, 28,658 bytes
shopify.com	12,801 bytes	yes	yes	12	72	real, 19,144 bytes

All three start with an H1, the only element llmstxt.org requires, and all three have a blockquote summary. github.com serves a file of the same size at both addresses; shopify.com serves a longer one at /llms-full.txt. The data does not say whether anyone reads these files.

The two pages that look like files

forbes.com redirected both requests to a consent page (/consent/ketch/?toURL=...) of 4,945 bytes. reddit.com answered both with a web page of about 8.4 KB at the requested address. For a client that checks only the status code, both look like a published file.

What could not be measured

Six sites sit outside the 22. reuters.com answered 401, wsj.com, medium.com and tripadvisor.com 403, imdb.com 202, and washingtonpost.com did not answer in time. They may or may not publish a file. The data cannot say, and a different client, or the same client at another time, might get a different answer.

Rather have this done for you?

Send the site address and what you want to know. We run the audit and send the report.

Send a request

Passes differ

The same list was requested again with curl, which classifies with separate code. 51 of 56 answers match; the 5 that differ come from 3 sites:

theguardian.com. curl over HTTP/2 gets a 200 page of 3,038 bytes for both files. curl with --http1.1 and the script, which speaks HTTP/1.1, get a 404. Three repeats gave the same split, so the answer depends on the HTTP version.
bloomberg.com. The script got the file at 21:47:50 UTC. curl got a 403 about 4 minutes later, and so did the script and curl in later tries within the hour. The cause is not known.
techcrunch.com. /llms-full.txt answered 403 to the script and 404 to curl. A one-off.

An earlier pass of the same list the same day matched the CSV for 27 of 28 sites on /llms.txt: stackoverflow.com answered 403 then and 404 now. The CSV is one run, not a consensus of runs.

Limits

Each site was requested once per file, from one IP address, with one User-Agent. Bot protection decides on more than the User-Agent and answered one client differently within minutes.
The 28 sites were chosen by the author: large publishers, marketplaces and reference sites. They are not a sample of the web, and no share here describes the web.
real says the file starts like markdown. It does not say the file is useful, complete or read by anyone.
Only /llms.txt and /llms-full.txt at the root were requested. Files in subfolders and /.well-known/llms.txt are not covered.
/llms-full.txt is a convention. The llmstxt.org text does not define it.
Files change. The data describes 2026-11-08, between 21:45 and 21:50 UTC.

CSV columns

Column	Values	Meaning
`site`, `checked_at_utc`	domain, ISO time	Requested domain and the time of its two requests
`llms_`, `full_`	see below	The `/llms.txt` and the `/llms-full.txt` request
`user_agent`, `note`	text	The User-Agent sent; a retry on `www.`, error texts

Inside llms_ and full_: final_url, class, http_status, content_type, bytes, then spec (ok when an H1 comes first), blockquote, h2 and links, filled for real files only.

Re-run it

bash

python3 check_llms_txt.py --selftest
python3 check_llms_txt.py sites.txt --out my-run.csv

The script needs Python 3.8 or newer and nothing else. The run takes about 4 minutes. If you publish your own run, replace the contact in the USER_AGENT constant with yours.

Cite it

The CSV and this page are licensed CC BY 4.0, the script MIT. Name the source and link to this page. Suggested form:

ADLAB, "llms.txt on 28 large sites: who publishes a real file (data, 2026-11-08)", https://adlab.ee/en/resources/ai-search/llms-txt-27-sites, CC BY 4.0.