Throttle AI crawlers in nginx instead of blocking them

What each class gets

The file sorts User-Agents into three classes and gives each its own allowance. Anything that matches no pattern, Googlebot and Bingbot included, has an empty key and is never counted.

The grouping is the author's, built from the operators' own crawler pages on 2026-11-08; the sources for every bot are in the reference table of the robots.txt resource. CCBot, Bytespider and anthropic-ai sit in the training class by the author's call: the operators' pages do not tie them to training.

Install

sudo cp nginx-ai-bots-throttle.conf /etc/nginx/conf.d/ai-bots-throttle.conf
sudo nginx -t && sudo nginx -s reload
sudo tail -f /var/log/nginx/ai-bots-throttle.log

The stock nginx.conf includes conf.d/*.conf inside http { }, so the file covers every server on that nginx. It needs nginx 1.17.6 or newer, because the rejection log and the Retry-After header read $limit_req_status.

One caveat: add_header and access_log are not inherited into a server or location that defines its own. If yours does, repeat the Retry-After and access_log lines there. nginx 1.29.3 added add_header_inherit merge; for the headers; it was not tested here.

Run it as a dry run for the first week. Uncomment limit_req_dry_run on; (available since nginx 1.17.1), reload, and read the log: lines marked REJECTED_DRY_RUN are the requests the live config would have refused. Nothing is refused yet.

How the config works

Three directives do the work. The map turns a User-Agent into the bot's name, the zone counts requests per name, and limit_req applies the zone.

map $http_user_agent $ai_bot_train {
    default "";
    ~*ClaudeBot   ClaudeBot;
    ~*GPTBot      GPTBot;
    # ... the rest of the training class
}
limit_req_zone $ai_bot_train zone=ai_train:1m rate=1r/s;
limit_req      zone=ai_train burst=10 nodelay;
limit_req_status 429;

The nginx documentation says that requests with an empty key value are not accounted. A visitor whose User-Agent matches no line gets the key "", so nothing is counted for it. The test confirms it: 30 quick requests as Googlebot, as Bingbot, as Chrome and with no User-Agent at all all returned 200.

burst=10 nodelay means the first request and ten more pass at once, which makes 11, and after that the bot gets one request per second. The test measured it: with the bucket drained and a 3 second wait, 10 requests returned 3 × 200 and 7 × 429. Retry-After: 10 is the time a full burst needs to refill at 1 request per second.

The key is the bot name, not the IP address. All addresses of ClaudeBot share one allowance, so "1 request per second" holds for the whole bot. GPTBot has its own bucket: in the test it got 11 × 200 right after ClaudeBot had been refused.

Do the bots slow down on a 429?

None of the operator pages used for this resource says how its crawler reacts to a 429 or to Retry-After. What they do document is Crawl-delay: Anthropic says it supports the non-standard extension, and the Common Crawl FAQ says CCBot obeys it. Amazon says Amazonbot does not support it, and Apple says Applebot does not follow it. Google does not support it.

So the plan has two parts. Add Crawl-delay in robots.txt for ClaudeBot and CCBot, and keep the throttle for the rest. After a week, count the refusals per bot:

awk -F'"' '/REJECTED/ {print $2}' /var/log/nginx/ai-bots-throttle.log | sort | uniq -c | sort -rn

A bot that backs off shows fewer refusals in week 2 than in week 1. A bot with the same count in both weeks ignores the 429 and its Retry-After: 10, and the throttle then only protects your server.

Pick the numbers from your own log

The defaults (1 r/s for training bots, 5 r/s for the other two classes) are starting values chosen by the author, not vendor recommendations. To see what ClaudeBot does today, count its requests per second in the access log:

grep -i claudebot /var/log/nginx/access.log | awk '{print $4}' | sort | uniq -c | sort -rn | head -5

The first line is the busiest second. If ClaudeBot peaks at 9 requests per second and you are willing to serve 2, set rate=2r/s. The log audit kit gives the same view per bot and per URL.

Why Googlebot and Bingbot are not in the list

Google's page on reducing crawl rate names 500, 503 and 429 as the status codes for slowing Googlebot down, and limits the advice to 1 to 2 days. It adds that if Googlebot sees these codes on the same URL for several days, the URL may be dropped from the index. A permanent throttle on Googlebot is that situation. Google's alternative is a special request filed through Search Console, which it says can take several days to evaluate. Only Google's documentation was checked for this resource; Bingbot is left out by the same reasoning, not by a Bing source.

Sources

Checked on 2026-11-08.