script

llms.txt checker: a real file, or a 200 page that only looks like one?

Python script, stdlib only: sorts /llms.txt and /llms-full.txt into real, html200, 404, blocked or error, and checks the llmstxt.org format.

v1.0
updated 8 November 2026
MIT
Vladislav Krivorutško

Download check_llms_txt.py Download all (.zip)Jump to the text

5 of 28
large sites answer /llms.txt with 200
3
of those send a real file, 2 send a web page
51 of 56
answers got the same class from curl
0
packages to install

A status of 200 on /llms.txt does not mean the site has the file. On 2026-11-08, 5 of 28 large sites answered /llms.txt with HTTP 200, and only 3 of them sent a real file. forbes.com and reddit.com sent a web page. This script requests /llms.txt and /llms-full.txt, reads the body and tells these cases apart.

It is one Python file with no packages. Give it domains, get a table and a CSV. It sends one honest request per file and does not try to get around a block: a 401, 403, 202 or 429 is recorded as blocked, which means "not measurable from here". The data from the run on 28 large sites is a separate resource: llms.txt on 28 large sites.

Files, grouped by job

4 files · or all in one zip

Read me

1 file

README.md
3-step run, class rules, CSV columns, limits
4.2 KB

Run it

2 files

check_llms_txt.py
The checker: Python 3.8+, standard library only, with an offline self-test
16 KB
curl_check.sh
Cross-check: the same classes from curl and shell, separate code
3.1 KB

Try it first

1 file

sites.txt
The 28 public domains of the ADLAB dataset, one per line
583 B

Quick start

Unzip the download and check Python: python3 --version (3.8 or newer).
Run the offline self-test, then your domains:
bash
```
python3 check_llms_txt.py --selftest
python3 check_llms_txt.py example.com
python3 check_llms_txt.py sites.txt --out result.csv
```
Give domains as arguments or a file with one domain per line. Each domain gets two GET requests, 2 seconds apart.
Read the table, or open the CSV. To check a draft before it goes live: python3 check_llms_txt.py --file llms.txt.

What the classes mean

Class	Rule
`real`	HTTP 200, the body is not an HTML page, and the text starts with `#` (after an optional BOM)
`html200`	HTTP 200, but the body is an HTML page: a soft 404, a consent page or a redirect to the home page
`404`	HTTP 404 or 410
`blocked`	HTTP 401, 403, 202 or 429, or a redirect to a page whose path says `blocked`, `captcha`, `challenge` or `denied`
`error`	No HTTP answer: DNS, TLS, timeout, reset, more than 3 redirects
`other`	A 200 that is not HTML but does not start with `#`, an empty body, a 5xx

The script follows up to 3 redirects, accepts gzip, times out after 20 seconds and sends the User-Agent ADLAB-research (+https://adlab.ee). If the bare domain gives no HTTP answer, it retries once on the www. host and writes that into the note column. If you publish your own run, put your own contact into the USER_AGENT constant.

Format check

For a real file the script checks the format of llmstxt.org, version 2, modified 2026-08-10. The text lists an H1 with the name of the site first and says of it: "This is the only required section". The script reports:

spec: ok when the first non-blank line is an H1, no_h1 otherwise;
blockquote: whether a > summary appears before the first H2 (optional in the text);
h2 and links: the number of H2 sections and of list items that hold a link.

/llms-full.txt is a common convention for one long file. The llmstxt.org text does not define it, and the script applies the same rules to it.

Example output

The run on 2026-11-08, nine of the 28 rows and the totals:

text

site                /llms.txt         /llms-full.txt    spec of llms.txt
nytimes.com         404 (404)         404 (404)
washingtonpost.com  error             error
wsj.com             blocked (403)     blocked (403)
bloomberg.com       real (200)        404 (404)         ok, quote=true, h2=27, links=141
forbes.com          html200 (200)     html200 (200)
shopify.com         real (200)        real (200)        ok, quote=true, h2=12, links=72
github.com          real (200)        real (200)        ok, quote=true, h2=12, links=117
imdb.com            blocked (202)     blocked (202)
zillow.com          404 (404)         404 (404)

/llms.txt: 404 17, blocked 5, error 1, html200 2, real 3  (of 28)
/llms-full.txt: 404 17, blocked 6, error 1, html200 2, real 2  (of 28)

Rather have this done for you?

Send the site address and what you want to know. We run the audit and send the report.

Send a request

Cross-check with curl

curl_check.sh sorts the same answers with curl and shell only, written separately from the Python code:

bash

bash curl_check.sh sites.txt > curl-result.tsv

Run both tools back to back and compare the classes. On 2026-11-08, 51 of 56 answers matched, and the 5 that did not come from 3 sites. theguardian.com answered a 200 page to curl over HTTP/2 and a 404 to curl over HTTP/1.1 and to the script. bloomberg.com sent the file to the script and a 403 to curl 4 minutes later, then a 403 to both. One /llms-full.txt request on techcrunch.com got a 403 once and a 404 otherwise. A difference between two tools is a finding, not a bug to hide: the details are on the data page.

Who it is not for

Anyone who needs a verdict that a site has no llms.txt. blocked and error mean the client could not look, and some sites answer differently by time or HTTP version.
Sites behind a login or a bot wall that serves a challenge page with a 200. The script catches block pages that arrive by redirect to a path named blocked, captcha, challenge or denied, and nothing more.
Anyone who wants to know whether AI products read the file. The script looks at what a server answers, not at who asks. For that question, use llms.txt in your server logs.

Gotchas

One request is one data point. Bot protection decides on more than the User-Agent. Bloomberg's answer changed within minutes on 2026-11-08.
HTTP version matters. Python's urllib speaks HTTP/1.1, curl defaults to HTTP/2 where the server offers it, and one large site answered the two differently.
A consent wall is html200. forbes.com redirected to a consent page with a 200.
real is a claim about the first byte. The file starts like markdown. It can still be outdated or empty of useful links.
Only the root paths. /.well-known/llms.txt and files in subfolders are not requested.
Bodies are read up to 5 MB. Larger files are measured up to that limit, and the note column says so.

Tested on

2026-11-08: Ubuntu 24.04 (WSL2), Python 3.12.3, bash 5.2.21, curl 8.5.0. From a clean folder holding only the files of this download: the self-test passed 24 of 24 checks, and the run on the 28 domains of sites.txt took 3 min 41 s. The self-test also passed on Windows with Python 3.13.7. The local-file mode was checked on a draft file from a sandbox site. The live run was not tested on macOS or on Windows, and not from other IP addresses or countries.