On 2026-11-08, 18 of 27 large sites closed at least one of 11 AI crawlers completely in robots.txt, and 9 closed none. ClaudeBot and Bytespider were closed most often, by 16 sites each. GPTBot and CCBot followed with 14 each, PerplexityBot with 11, and Google-Extended was closed by 6. bbc.com and bloomberg.com closed all 11.
- dataset
How 27 big sites treat AI crawlers in robots.txt (data, 2026-11-08)
27 large sites checked on 2026-11-08: who closes ClaudeBot, GPTBot, CCBot or PerplexityBot completely in robots.txt. CSV, script and method, CC BY 4.0.
- v1.0
- updated 8 November 2026
- CC-BY-4.0
- Vladislav Krivorutško
18 of 27
large sites close at least one AI crawler
9
close none
16
sites close ClaudeBot and Bytespider
11
AI crawlers checked
Files in this resource
4 files · or all in one zipai-bots-robots-27-sites.csv
One row per site: 11 AI crawler tokens, true when the robots.txt closes the bot with Disallow: /
scan_robots.py
Python 3.8+, standard library only; reproduces the measurement for any list of domains
sites.txt
The 28 domains that were requested
README.md
Definition, CSV columns, how to re-run, limits
What was measured
The script requested /robots.txt from 28 domains between 17:45:44 and 17:46:48 UTC on 2026-11-08, one request per domain, with the User-Agent ADLAB-research (+https://adlab.ee) and a 2 second pause between domains. 27 domains returned a file. stackoverflow.com answered HTTP 418 and is in the CSV as an error, outside the 27. The script does not try to get around that.
A site counts as closing a bot when its robots.txt has a group whose User-agent line names exactly that bot, ignoring case, and the group contains Disallow: /. A User-agent: * group does not count, and neither does a partial rule such as Disallow: /news/. Several User-agent lines in a row share one group, comments and blank lines are skipped, and two groups for the same bot are merged. The 11 tokens are those in the table below.
Results by bot
| Bot | Operator | Class per the operator's page | Sites closing it, of 27 | Share |
|---|---|---|---|---|
| ClaudeBot | Anthropic | training | 16 | 59% |
| Bytespider | ByteDance | not documented | 16 | 59% |
| GPTBot | OpenAI | training | 14 | 52% |
| CCBot | Common Crawl | open dataset, AI training not addressed | 14 | 52% |
| meta-externalagent | Meta | training and product indexing | 13 | 48% |
| Amazonbot | Amazon | may be used to train Amazon AI models | 12 | 44% |
| anthropic-ai | Anthropic | not in the current article | 11 | 41% |
| PerplexityBot | Perplexity | search | 11 | 41% |
| ChatGPT-User | OpenAI | user-triggered | 8 | 30% |
| OAI-SearchBot | OpenAI | search | 7 | 26% |
| Google-Extended | control token for training and grounding | 6 | 22% |
The class column comes from the operators' own pages, collected with their links in the robots.txt for AI bots resource. Where the cell says "not documented", no statement was found on 2026-11-08.
What the data shows
Training crawlers are closed more often than the search and user-triggered bots of the same operator. GPTBot is closed on 14 sites, ChatGPT-User on 8 and OAI-SearchBot on 7. All 7 sites that close OAI-SearchBot also close GPTBot, and 7 sites close GPTBot without closing OAI-SearchBot.
Google-Extended is the least closed of the 11, on 6 sites. All 6 also close ClaudeBot, and 10 sites close ClaudeBot without Google-Extended. The data does not say why.
Four sites publish User-agent: * with Disallow: / and close none of the 11 bots by name: reuters.com, wsj.com, reddit.com and imdb.com. Their files tell every crawler that is not named elsewhere to stay out, so for them the counts understate the exclusion. quora.com has the same * rule and closes 6 bots by name.
Ten sites name at least one of the 11 bots in a group that has no Disallow: /. forbes.com closes 8 bots and names OAI-SearchBot in a group with Allow: /. github.com closes only Bytespider and names five other bots in groups without Disallow: /. The column named_not_closed in the CSV lists these cases.
Nine sites close none of the 11: reuters.com, wsj.com, wikipedia.org, reddit.com, walmart.com, shopify.com, nerdwallet.com, booking.com and imdb.com.
Rather have this done for you?
Send the site address and what you want to know. We run the audit and send the report.
Limits
- A robots.txt is a statement, not proof. Firewalls, WAF rules, rate limits and IP blocks are invisible to this measurement, and a site that closes a bot in the file may also serve it pages, or the other way round.
- The 27 sites were chosen by the author: large publishers, marketplaces and reference sites. They are not a sample of the web, and no percentage here describes the web.
- Each site was fetched once, from one IP address, with one User-Agent. A site may serve a different file to other visitors.
- Only
Disallow: /counts. A site that closes/news/to GPTBot shows up asfalse. - Files change. The data describes 2026-11-08, between 17:45 and 17:47 UTC.
- An earlier pass of the same list on the same day counted forbes.com as closing none of the 11 bots. forbes.com sends its robots.txt gzip-compressed, and the script now unpacks it. The README records the difference.
CSV columns
| Column | Values | Meaning |
|---|---|---|
site, scanned_at_utc | domain, ISO time | Requested domain and the time of its request |
result, http_status, bytes | file, no_file, error; number; number | Whether a robots.txt came back, the last HTTP status, the decoded size |
any_bot_closed | true, false, empty | At least one of the 11 bots is closed |
GPTBot to meta-externalagent | true, false, empty | true = closed with Disallow: /; empty when no file came back |
star_disallow_all, named_not_closed | true, false; bot names | The * group closes everything; bots named in a group without Disallow: / |
Re-run it
bash
python scan_robots.py sites.txt --out my-scan.csv
python scan_robots.py example.com --bots GPTBot ClaudeBot Google-Extended
python scan_robots.py --selftest
The script needs Python 3.8 or newer and nothing else. The self-test runs 13 parser checks without network access: stacked User-agent lines, case, comments, a byte order mark, separate groups for one bot, Disallow: with an empty value, and a gzip-compressed body. It sends one request per site and waits 2 seconds between sites. If you publish your own run, replace the contact in the USER_AGENT constant with yours.
Cite it
The CSV and this page are licensed CC BY 4.0, the script MIT. Name the source and link to this page. Suggested form:
ADLAB, "How 27 big sites treat AI crawlers in robots.txt (data, 2026-11-08)", https://adlab.ee/en/resources/ai-search/ai-bots-robots-27-sites, CC BY 4.0.
Files are provided as is: run them on your own data and check the result.
Want an audit like this on your site?
If you would rather have it done for you, send a request and describe the site.