llms requests in the synthetic sample, from 19 addresses
7
kinds of client, split by IP check
3
file locations counted
0.2 s
for the report on the sample, after the kit run
On the synthetic sample log, the report finds 54 requests for llms files from 19 addresses in 7 days. 7 of them come from AI bots whose IP is confirmed, 11 use an AI bot's name from an address outside its list, 13 come from SEO tools and 18 from browsers and scripts. The sample is invented to show how the report reads: its numbers say nothing about how often real bots fetch llms.txt.
This is an add-on to the server log audit kit. The kit already sorts every request by bot name and checks the source IP against the operator's official ranges. This SQL file reuses that work to answer one question for /llms.txt, /llms-full.txt and /.well-known/llms.txt: who asks for them, and what does your server answer. The same clients' requests for robots.txt and sitemaps sit next to the llms counts as a control. It runs on your machine with clickhouse-local; nothing is uploaded.
Files, grouped by job
4 files · or all in one zip
Read me
1 file
README.md
What you need, the 3-step run, sections, limits
8.9 KB
Run it
1 file
llms_report.sql
8 sections over the kit's enriched.parquet: by kind of client, by bot, by day, with robots.txt as a control
6.1 KB
Try it first
2 files
sample/make_llms_sample.py
Generator of the synthetic log (deterministic, stdlib only)
10 KB
sample/llms.sample.log
Synthetic log of 1,127 lines for example.com, 7 days, with llms requests from every kind of client
228 KB
Quick start
Run the kit on your logs, or on the sample from this download. The sample is synthetic and uses example.com:
bash
cd log-audit-kit
SKIP_SITEMAP=1 bash run.sh -d example.com -i 203.0.113.10 ../llms-txt-in-your-logs/sample/llms.sample.log
For your own logs: bash run.sh -d yourdomain.com -i YOUR,OWN,IPS /var/log/nginx/access.log*. Pass your own server and monitor IPs with -i, or your checks count as clients.
Run the report on the kit's work folder:
bash
cd ../llms-txt-in-your-logs
OUT=$PWD/../log-audit-kit/log-audit-example.com
sed "s#__OUT__#$OUT#g" llms_report.sql | clickhouse local --multiquery > llms_report.txt
Read llms_report.txt, then delete the kit's work folder: it holds visitor IPs.
The report needs enriched.parquet, which run.sh makes. A raw logs.parquet has no bot names and no IP check, so it is not enough. The kit gives every column this file reads (bot, cls, verified, kind, np), so nothing in the kit has to change.
Kinds of client
Kind
Rule, from the kit's columns
1 AI bot, IP confirmed
AI class (ai-training, ai-search, ai-user) and verified is true
2 bot name, IP outside its list (fake)
verified is false: the User-Agent claims a bot, the IP is not in the operator's list
3 AI bot name, no IP list to check
AI class, verified empty: Meta, Amazon, ByteDance and Common Crawl have no list in the kit
4 search engine bot
Google, Bing, Apple, Yandex and other search engines
5 SEO tool
AhrefsBot, SemrushBot, Screaming Frog and similar
6 browser User-Agent
An ordinary browser string: a person, or a bot that hides
Requests from the addresses you pass with -i are left out.
Sections
Section
Answers
0
Window, total requests, how many were for an llms file
1
Requests by kind of client, file and response (200, 3xx, 404)
2
The same by bot name, with first and last day
3, 4
Control: pages, robots.txt, sitemaps and llms files for the same clients and bots
5
Responses by day: a file published on day 4 shows as 404, then 200
6
User-Agents of browsers and scripts that asked for an llms file
7
Confirmed AI bots by class: pages, robots.txt and llms files
Sample output (synthetic)
Sections 1, 3 and 7 from the run on 2026-11-08. The site in the sample publishes /llms.txt on day 4, so some of the 404s are requests that came before the file existed (section 5 shows it).
A User-Agent is not proof. Rows 2 and 3 exist because anyone can send ClaudeBot in a header. Only row 1 is a bot whose IP the operator lists as its own.
Compare with robots.txt. Section 3 shows the same clients on robots.txt. A bot class with many robots.txt requests and no llms requests reads your rules and has not asked for the file in your window. Neither number says what the bot does with them.
Look at the class of AI bot. User-triggered fetchers such as ChatGPT-User act when a person asks an assistant about a page; training and search crawlers work on their own schedule. The operators' pages describing each class are collected in robots.txt for AI bots. A few llms requests from a user-triggered class mean people sent the assistant to the file, not that a crawler found it.
A request is not use. A 200 means the server sent the bytes. The log cannot show whether a model read them.
Check the day. A file that did not exist on the first days answers 404 to everybody. Section 5 separates those days.
Empty is a result. If section 1 has no row for kind 1, no confirmed AI bot asked for an llms file in your window.
Who it is not for
Sites that write no access logs (Vercel, Cloudflare Workers): there is nothing to read.
Sites whose CDN answers /llms.txt from cache. On Cloudflare, check cf-cache-status: HIT means the request never reached your server log.
Logs in a format other than nginx or Apache combined, unless you add a build_<name>.sql to the kit first.
Gotchas
Trailing slash. The kit's np column drops it, so /llms.txt/ counts as /llms.txt. In the sample the two such requests are 301s.
Case. The report lower-cases the path, so /LLMS.txt is counted too.
A file in a subfolder./docs/llms.txt is allowed by the llmstxt.org text and is counted in the subfolder column.
Your own checks. Without -i, your monitor and your curl tests show up as clients. The sample has four such requests from 203.0.113.10, which the report leaves out.
IP lists move.verified depends on the operators' lists as of the kit run (see Official crawler IP ranges).
Tested on
2026-11-08: Ubuntu 24.04 (WSL2), bash 5.2.21, ClickHouse 26.10.1.2121. From a clean folder holding only the kit and this download: the kit ran on the sample log of 1,127 lines in 6.1 s, including the download of the IP ranges, and the report took 0.2 s. All 1,127 lines parsed. The report was not run on a production log, not on logs with millions of lines, and not on macOS. The sample's counts of llms requests are invented; the classification of its addresses (confirmed, fake, no list) comes from the real check against the operators' lists on that date.
