To opt out of AI training and stay in AI search answers, disallow GPTBot, ClaudeBot, Applebot-Extended, meta-externalagent, Amazonbot and MistralAI-Training, and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. OpenAI's documentation says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot may reduce a site's visibility in its search results. The download holds four ready variants and a table of 30 bots, each row pointing to the operator's own page.
- template
- config
robots.txt for AI bots: block training, keep search
Copy-paste robots.txt variants for AI crawlers: opt out of training, close everything, or block single sections. Reference table of 30 bots with sources.
- v1.0
- updated 8 November 2026
- MIT
- Vladislav Krivorutško
4
classes of AI bot
4
ready variants, A to D
104
assertions passed in the test
2
control tokens that are not crawlers
Files in this resource
2 files · or all in one zipFour classes of AI bot, and what a Disallow does to each
One company runs several bots with different jobs: OpenAI alone documents GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot as four separate bots with separate settings.
| Class | What the bot does | Examples | What a Disallow does, per the operators' pages |
|---|---|---|---|
| Training | Collects pages for model training or datasets | GPTBot, ClaudeBot, meta-externalagent, MistralAI-Training | OpenAI: content should not be used for training. Anthropic: future material is excluded from training datasets |
| Search | Indexes pages so a search product or assistant can cite them | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot | OpenAI: not shown in ChatGPT search answers. Anthropic: may reduce visibility and accuracy in search results |
| User-triggered | Opens one page because a person asked an assistant about it | ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher | OpenAI: rules may not apply. Perplexity: generally ignored. Meta: may bypass. Amazon: may not follow all directives |
| Control token | Not a crawler; a robots.txt name that decides how already crawled content is used | Google-Extended, Applebot-Extended | Google: no effect on Search inclusion or ranking. Apple: pages can still appear in search results |
In the CSV the two control tokens are marked training. Seven rows are marked other: CCBot, Bytespider, anthropic-ai, GoogleOther, Google-CloudVertexBot, OAI-AdsBot and facebookexternalhit.
Pick a variant
| Variant | What it does | What you give up | Fits |
|---|---|---|---|
| A, open | Every crawler follows your * group | Nothing is blocked, training crawlers read everything | Sites with no objection to training |
| B, block training | Disallow for 7 training and control tokens | Google-Extended also covers grounding in Gemini Apps | Sites that want to be cited by assistants and object to training |
| C, block all | Disallow for every AI crawler in the list | Per OpenAI and Anthropic, visibility in their search answers | Licensing or legal reasons to stay out of all AI products |
| D, by section | Disallow for named paths only | The shared * rules must be repeated for each named bot | A free blog next to a paid area |
Variant A needs no rules. The template says so on purpose: a GPTBot group with Allow: / would replace your * group for GPTBot and open /admin/ and /cart/ to it.
Variant B, ready to paste
text
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Google-Extended
Disallow: /
Stacked User-agent lines share the rule under them. Everything not named, OAI-SearchBot and the other search and user-triggered bots included, keeps following your * group.
Three more names are optional: CCBot, Bytespider and anthropic-ai. They are optional because the operators' pages do not tie them to training. Common Crawl describes an open dataset and does not address AI training, ByteDance has no operator page that could be found on 2026-11-08, and Anthropic's current crawler article does not mention anthropic-ai.
Delete the Google-Extended line if you want to stay citable in Gemini Apps, for the reason in the next section.
Google-Extended and Applebot-Extended are not crawlers
Google writes that Google-Extended has no HTTP user agent string of its own; crawling uses Google's existing agents, and the robots.txt token only sets how content may be used. The token covers training of future Gemini models and grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. Google states that it does not affect inclusion in Google Search and is not a ranking signal.
Google's page on AI features says a page must be indexed and eligible to show with a snippet to appear as a supporting link in AI Overviews or AI Mode. For limiting what Search shows from your pages it names nosnippet, data-nosnippet, max-snippet and noindex. The same page mentions Google-Extended only as the control for AI training and grounding in other Google systems. Google's crawler page adds that it intends to support the AIPREF training-preference signal in early 2027.
Apple's page says Applebot-Extended does not crawl; it only determines how data crawled by Applebot is used. Pages that disallow it can still be included in search results, and its rules are not considered in ranking.
What closing everything costs
Variant C is the one that closes the whole site to AI search products. These are the operators' own statements on 2026-11-08:
| Operator | Bot | What the operator's page says |
|---|---|---|
| OpenAI | OAI-SearchBot | Opted-out sites will not be shown in ChatGPT search answers; settings for GPTBot and OAI-SearchBot are independent |
| Anthropic | Claude-SearchBot, Claude-User | Blocking may reduce visibility and accuracy in user search results, and visibility for user-directed web search |
| Perplexity | PerplexityBot | Recommends allowing it so the site appears in Perplexity search results |
| Apple | Applebot | Allowing it lets content appear in search results in Spotlight, Siri and Safari |
| Meta | Meta-WebIndexer | Allowing it helps Meta cite and link to your content |
| Amazon | Amzn-SearchBot | Allowing it makes content eligible to appear in search in products such as Alexa |
Opinion, not a documented fact: if being named in ChatGPT, Claude or Perplexity answers is the goal, B and D keep it and C removes it. If the goal is to stay out of everything for licensing reasons, C matches the goal, but robots.txt will not stop the user-triggered fetchers that ignore it. OpenAI, Perplexity, Meta, Amazon and Google each say so on their own pages. For those you need a firewall rule, and the nginx throttle is the softer alternative.
Rather have this done for you?
Send the site address and what you want to know. We run the audit and send the report.
Close one section only (variant D)
A crawler that has its own group ignores the * group. Google's documentation says it uses the group with the most specific matching user agent, and RFC 9309 (section 2.2.1) describes the same matching. This version looks right and is not:
text
User-agent: *
Disallow: /admin/
User-agent: GPTBot
Disallow: /members/
GPTBot may fetch /admin/ here: it follows only its own group, which holds 1 rule, and /admin/ is not in it. Repeat the shared rules:
text
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /admin/
Disallow: /members/
Disallow: /guides/
Allow: /guides/free/
The longest matching path wins, so /guides/free/ stays open inside the closed /guides/. For rules of equal length, Allow wins under RFC 9309; Google says it uses the least restrictive rule.
Gotchas
- RFC 9309 states that robots.txt rules are not a form of access authorization. A bot that ignores the file needs a firewall or WAF rule.
- Google does not support
Crawl-delay. Anthropic says it supports it for its bots and the Common Crawl FAQ says CCBot obeys it, while Amazon and Apple say Amazonbot and Applebot do not. - Changes are not instant. OpenAI and Perplexity say up to about 24 hours, Meta says its crawlers may cache the file for up to 24 hours, Google caches it for up to 24 hours, and DuckDuckGo says 72 hours for DuckAssistBot.
- A matching User-Agent proves nothing; anyone can send it. The operators publish IP ranges as JSON (OpenAI, Anthropic, Perplexity, Apple, Common Crawl, DuckDuckGo, Mistral, Google), and the CSV lists the URLs. Anthropic adds that blocking by IP may not guarantee an opt-out, and names robots.txt as the method.
- A robots.txt that cannot be fetched is dangerous. RFC 9309 says a crawler must assume complete disallow when the file is unreachable (5xx), and Google stops crawling for the first 12 hours of a failed fetch.
- Google reads the first 500 KiB of the file and ignores the rest.
- Anthropic describes a block as a signal for future materials. None of the pages checked say what happens to content collected before the block: not documented.
Check the change
bash
curl -sI https://example.com/robots.txt | head -n 5
curl -s https://example.com/robots.txt | grep -n -i -A1 'GPTBot'
Expect a 200 status and content-type: text/plain. Then open the robots.txt report in Search Console: it shows the fetch status, the time of the last check, the file size and the number of parsing issues, and offers a recrawl request for a fixed file.
After 24 hours, read your access log for hits after the change:
bash
grep -i 'GPTBot' /var/log/nginx/access.log | grep -v 'robots.txt' | tail -n 5
A line dated after the change means the User-Agent kept crawling. It does not yet mean the real GPTBot did: compare the source IP with OpenAI's published list. The log audit kit runs that comparison for all the bots in the table.
Who this does not fit
- You need to prevent copying. robots.txt is a request, and the template cannot enforce it.
- You want deletion of content collected earlier. None of the operator pages checked describe a way to do that through robots.txt.
- You want to opt out of Gemini training but stay in Gemini Apps answers. Google documents one token, Google-Extended, for training and grounding together, and no separate token was found.
Tested on
On 2026-11-08 each variant was enabled in turn and parsed with Python 3.13 urllib.robotparser and with a longest-match checker written to RFC 9309. All 104 assertions passed: every token in B, C and D resolves as the tables above say, Googlebot and Applebot stay open in C, /guides/free/ stays open in D, and /admin/ stays closed for GPTBot in D. Google's own parser was not used; run the Search Console report after you deploy.
Sources
| Operator | Page | Checked |
|---|---|---|
| OpenAI | Crawlers and user agents | 2026-11-08 |
| Anthropic | Does Anthropic crawl data from the web | 2026-11-08 |
| Perplexity | Perplexity crawlers | 2026-11-08 |
| Common crawlers, user-triggered fetchers, AI features, robots.txt | 2026-11-08 | |
| Apple | About Applebot | 2026-11-08 |
| Meta | Web crawlers | 2026-11-08 |
| Amazon | Amazonbot | 2026-11-08 |
| Mistral | Mistral crawlers | 2026-11-08 |
| Common Crawl | CCBot, FAQ | 2026-11-08 |
| DuckDuckGo | DuckAssistBot | 2026-11-08 |
| IETF | RFC 9309, Robots Exclusion Protocol | 2026-11-08 |
| Search Console robots.txt report | 2026-11-08 |
Operators change these pages. Where a row in the CSV says "not documented", no statement was found on the page on that date.
Files are provided as is: run them on your own data and check the result.
Want an audit like this on your site?
If you would rather have it done for you, send a request and describe the site.