Topic · 3 resources
robots.txt templates, rate limits and measurements for the crawlers behind ChatGPT, Claude, Perplexity and Google's AI features.
27 large sites checked on 2026-11-08: who closes ClaudeBot, GPTBot, CCBot or PerplexityBot completely in robots.txt. CSV, script and method, CC BY 4.0.
What you get
18 of 27 large sites close at least one AI crawler
9 close none
16 sites close ClaudeBot and Bytespider
Copy-paste robots.txt variants for AI crawlers: opt out of training, close everything, or block single sections. Reference table of 30 bots with sources.
What you get
4 classes of AI bot
4 ready variants, A to D
104 assertions passed in the test
Tested nginx config: 429 with Retry-After when ClaudeBot, GPTBot or CCBot pass 1 request per second. Googlebot and Bingbot are never counted.
What you get
3 bot classes, each with its own allowance
1 r/s for training crawlers, burst 10
19 of 30 quick ClaudeBot requests got a 429
If you would rather have it done for you, send a request and describe the site.
