Short answer
On one of my own sites, AI bots made more than four times as many requests as Google. I took 30 days of server logs, 141,020 requests, and checked every bot against the IP ranges its operator publishes. Verified OpenAI and Perplexity bots made 7,356 requests. Verified Google bots made 1,747. With Anthropic's verified ClaudeBot added, the ratio is about 5.1.
Two more findings matter more than the ratio. A quarter of the requests that called themselves Googlebot were fake. And ChatGPT kept asking for pages that do not exist on the site. This is one young site of about 2,400 pages, so read it as a snapshot, not a verdict. The method below works on any site that writes access logs. The video above walks through the same audit, and the files for repeating it on your own logs are below the article.
Why a log file analysis shows what other SEO tools miss
A site crawler shows what exists on your site. Search Console shows what Google decided to crawl and index, and only for Google. The access log is the one record of what every bot actually requested and what your server answered: status code, bytes, user agent, IP address.
The test site is a price directory for local services: about 2,400 pages on Next.js behind Cloudflare. Cloudflare does not cache its HTML, so every page request reaches the origin server and lands in the log. That detail decides whether a log analysis is possible at all. If your CDN serves HTML from cache, those requests never reach your server. On Cloudflare, cf-cache-status: DYNAMIC on the homepage means the request went to the origin. Hosts that write no access logs, such as Vercel or Cloudflare Workers, leave nothing to analyse.
The audit ran locally with ClickHouse: 31 log files, 3.6 MB, 30.6 days from 8 October to 8 November 2026, every line parsed. Claude Code ran the steps and read the report, but the counting is plain SQL. I want numbers I can reproduce, not numbers a model produced.
Who was in the log
Less than half of the traffic looks like browsers, and that is not the same as people. On 2 November alone there were 12,743 browser-like requests.
| Class | Requests | Share |
|---|---|---|
| Browser-like clients | 61,523 | 43.6% |
| Scripts | 27,351 | 19.4% |
| SEO tools (Ahrefs 4,671, Semrush 3,011, DataForSEO 2,256) | 11,937 | 8.5% |
| Facebook link-preview bot | 9,553 | 6.8% |
| AI search bots | 7,303 | 5.2% |
| Empty user agent | 4,812 | 3.4% |
| AI training bots | 4,780 | 3.4% |
| My own server checks | 4,443 | 3.2% |
| Bing | 2,987 | 2.1% |
| AI user-triggered fetchers | 2,314 | 1.6% |
| Google, all requests that claim it | 2,278 | 1.6% |
| Apple | 1,361 | 1.0% |
All AI classes together come to 14,397 requests, 10.2% of the log. Google is 1.6%, and that figure still includes the fakes. Facebook's link-preview bot alone made more than five times as many requests as verified Google.
Why you cannot count Googlebot by user agent
A user agent is a text header. Any script can write "Googlebot" in it. Of the 2,278 requests that called themselves Google, 531 (23%) came from 21 IP addresses outside Google's published ranges. A count by user agent would have reported 2,278 Google requests and missed the split.
A second group was worse: 14 IP addresses kept switching names between ChatGPT-User, Claude-User, YouBot and Googlebot. They made 4,611 requests, and 73% ended in errors. Those are scanners wearing bot costumes. My rule since then: if a bot cannot be verified by IP, I do not count it as that bot. Google's own documentation tells every site, small or large, to verify Googlebot this way.
Who publishes the ranges, as of 8 November 2026:
| Operator | What is published | Effect on the audit |
|---|---|---|
| Google, Bing, Apple, OpenAI, Perplexity | JSON range files, per bot for OpenAI and Perplexity | Verified |
| Anthropic | One list for ClaudeBot, Claude-SearchBot and Claude-User, since 20 October 2026 | Verified |
| Common Crawl, DuckDuckGo, Mistral | JSON lists | Published, not loaded by my kit yet |
| Amazon | Three lists on its developer pages, not as files | Counted by user agent only |
| Meta, ByteDance | No list | Counted by user agent only |
In total, 10 operators publish 18 JSON lists with 3,084 entries. Google-Extended and Applebot-Extended have no IP ranges at all: they are robots.txt tokens, not crawlers. Two traps came up while I built the check. Six ranges sit in both OpenAI's GPTBot and OAI-SearchBot lists, so a lookup that stops at the first match marks real GPTBot requests as fake. And behind a CDN or load balancer the log must hold the real client IP (real_ip_header in nginx, mod_remoteip in Apache), otherwise every bot carries the proxy's address and looks fake.
What Googlebot did: the log against Crawl stats
Verified Googlebot made 1,708 requests in 30 days, about 56 a day, across 630 pages. The sitemap lists 2,420 URLs. Googlebot requested 623 of them, and 1,797 (74%) got no request at all. Among the provider pages, the largest section, 77% were never requested. About one Googlebot request in five (359) was for robots.txt.
Search Console tells the same story from Google's side. In Settings, Crawl stats for the last 90 days shows 98% of requests as "refresh", re-checks of pages Google already knows, and 2% as "discovery". HTML is 84% of the requests, Googlebot Smartphone made 84%, the average response time is 615 ms. Host status says the host "had problems in the past", while my log shows zero 5xx responses in the last 30 days. So that is history, but worth knowing about.
Crawl stats covers 90 days and the log covers 30, so I compare the structure, not the numbers. Both say the same thing: Google is not exploring this site, it is maintaining what it already has. My reading is that it is a young site and Google is careful with it. That is my interpretation, not something Google has said.
Is this a crawl budget problem? For a site of 2,400 pages, by Google's own scale, almost certainly not. I covered the criteria in when crawl budget actually matters. Here the 74% is a reason to check internal linking and the sitemap, not a sign of a starving site. "Not requested in the window" also does not mean "not indexed".
What AI bots fetched
There are three kinds of AI bot, and the difference decides what you should do with each. Search bots build the index behind AI answers. Training bots collect data for models. User-triggered fetchers open one page because a person asked an assistant a question.
| Bot | Class | Requests | Verified by IP | Note |
|---|---|---|---|---|
| OAI-SearchBot | search | 4,658 | 4,318 | 91% of requests to real sitemap pages |
| PerplexityBot | search | 1,945 | 1,687 | 1,536 on a single day, 25 October |
| Amazonbot | training | 1,764 | not checked | Amazon's lists are not loaded |
| ClaudeBot | training | 1,753 | 1,534 (87%) | 479 of its requests were for the sitemap |
| ChatGPT-User | user-triggered | 1,521 | 1,257 | Opens pages people ask about |
| GPTBot | training | 324 | 94 | 62% of requests ended in errors |
| Claude-User | user-triggered | 299 | 17 | Mostly scanners using the name |
OAI-SearchBot is the busiest: 3,658 requests to 143 service pages, about 26 per page in 30 days, so it re-reads each service page almost daily. PerplexityBot is not a polite guest: four fifths of its month came in one day. ClaudeBot fetched the same sitemap file 479 times. On this site the AI search bots look at the content more often than Googlebot does. Whether that turns into visibility in AI answers is a separate question, which I discuss in AEO vs GEO.
What ChatGPT opens, and the pages it asks for that do not exist
ChatGPT-User is the most useful bot in the log, because each request means a real person asked ChatGPT something that needed this site. After the homepage (685 requests), it opened specific service pages: windshield replacement and car painting 12 times each, home loans 11, tyre change 10. It comes for pages, not for guide files: in a month of logs from 839 sites, verified AI bots, ChatGPT-User included, requested llms.txt exactly once.
It also requested eight URLs that do not exist on the site: bathroom renovation, roof construction, waterproofing, gutter cleaning, attic insulation, insulation, joint renewal, and a tyre change page at the wrong address. ChatGPT guessed those addresses. For me that is a list of topics people ask about and for which I have no page yet. My rule for such 404s: if a page on that topic exists under another address, add a 301 to it. If the topic fits the site and there is no page, put it in the content plan. Otherwise leave the 404, and never redirect them all to the homepage. The difference between a 404, a 410 and a redirect is in what to do with deleted pages.
ChatGPT-User also requested my redirect path 221 times, although robots.txt disallows it. OpenAI says user-triggered fetches may ignore robots.txt. That is documented behaviour, not a bug.
What comes back is small. These are the visits in the same 30 days that arrived with an external referrer and landed on a page that answered without an error:
| Source | Visits | Distinct IPs |
|---|---|---|
| 120 | 25 | |
| Bing | 85 | 67 |
| ChatGPT | 41 | 20 |
| DuckDuckGo | 30 | 23 |
By distinct visitors Bing sent more people than Google here. Bots read thousands of pages, people arrive in dozens.
When AI bots overload a server
On a small site, load is not an issue. On one multilingual client site it was. Over 11 days a single crawler that called itself Meta's made 41,289 requests and pulled 2.4 GB, 13 times more requests than Googlebot's 3,055 on the same site. Meta publishes no IP list, so I can only say what the crawler claimed to be. About half of the AI training requests there went to redirects: waste for the bots and for the server.
The reflex of most developers I meet is to block the bots or throttle them hard. I understand it, but a bot you block cannot read your page, and an assistant that cannot read your page cannot cite it. OpenAI says it plainly: sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot may reduce visibility in its search results.
So I do it in this order:
- Verify by IP and drop the fakes. Part of the "AI load" is usually scanners using bot names.
- Separate the classes. Training bots can be slowed down or closed, search bots and user-triggered fetchers should get in.
- Fix the waste: redirects from your own links, parameter traps, heavy pages.
- Rate-limit instead of blocking. A bot that waits one second between requests makes at most 86,400 requests a day, whatever the size of the site.
The order matters: blocking first cuts off the search bots along with the crawler that caused the load. Untangling bot load this way is a regular part of my technical SEO audits.
robots.txt for AI bots: close training, keep search
To opt out of AI training and stay in AI search answers, disallow the training bots and the control tokens, and leave the search bots alone:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Google-Extended
Disallow: /
Stacked User-agent lines share the rule under them. OAI-SearchBot, Claude-SearchBot, PerplexityBot and everything else not named keep following your User-agent: * group. Three details catch people out:
- Google-Extended is not a crawler. Google says it does not affect inclusion or ranking in Google Search, but it does cover grounding in Gemini Apps. Delete that line if you want to stay citable there.
- A bot with its own group ignores the
*group. If you close one section for GPTBot only, repeat your shared rules (/admin/,/cart/) inside its group, or GPTBot gets access to them. - User-triggered fetchers may ignore robots.txt. OpenAI, Perplexity, Meta, Amazon and Google each say so about their own. For those, a firewall rule or a throttle is the tool.
The general rules of the file itself are in sitemap.xml and robots.txt best practices. The robots.txt template in the files has four variants (open, close training, close everything, close by section) and a table of 30 bots with each operator's own statement.
What do large sites do? On 8 November 2026 I checked the public robots.txt of 27 large publishers, marketplaces and reference sites. 18 close at least one of 11 AI bots completely (Disallow: / for that bot), 9 close none. ClaudeBot and Bytespider are closed by 16 sites each, GPTBot and CCBot by 14, PerplexityBot by 11, ChatGPT-User by 8, OAI-SearchBot by 7, and Google-Extended by only 6. Training bots are closed more often than search bots of the same operator: all 7 sites that close OAI-SearchBot also close GPTBot, and 7 more close GPTBot alone. The sites were chosen by me, so this is not a sample of the web. My own small directory blocks none of these bots. For a site that wants to show up in AI answers, I think that is the right call.
Throttling AI bots in nginx instead of blocking
The nginx config in the files sorts user agents into three classes and gives each its own allowance per bot name. Googlebot, Bingbot and ordinary visitors match no pattern and are never counted.
| Class | Allowance | 30 quick requests in the test |
|---|---|---|
| Training (ClaudeBot, GPTBot, CCBot and others) | 1 request per second, burst 10 | 11 × 200, 19 × 429 |
| Search (OAI-SearchBot, PerplexityBot and others) | 5 per second, burst 20 | 21 × 200, 9 × 429 |
| User-triggered (ChatGPT-User, Claude-User and others) | 5 per second, burst 20 | 21 × 200, 9 × 429 |
| Googlebot, Bingbot, Chrome, empty user agent | not counted | 30 × 200 |
A refused request gets 429 Too Many Requests with Retry-After: 10. I tested it in Docker on nginx 1.30.6 and 1.29.3. Run it as a dry run for a week first: nginx then only logs what it would have refused.
Three limits. No operator page I read says how its crawler reacts to a 429; Anthropic and Common Crawl document support for Crawl-delay, Amazon and Apple say their bots ignore it, Google does not support it. The throttle works on the user agent, so fakers spend the real bot's allowance; the throttle config notes how to key it by IP instead. And Googlebot stays out of the list on purpose: Google's guide on reducing crawl rate allows 429, 500 or 503 for one to two days only and warns that URLs answering like that for longer may drop from the index.
Server log thresholds: when to act
A number in the report is a reason to look, not a verdict. These are rules of thumb from three trial audits of production sites in October 2026, not Google thresholds. Google publishes no percentage for any of them.
| Signal | Act when | First step |
|---|---|---|
| Requests claiming Google that fail IP verification | any | Use verified hits only in every Googlebot figure |
| 5xx for verified Googlebot | above 1% of its requests | Error logs for those paths and times |
| Redirects for verified Googlebot | above 10% | Replace links in templates, menus and the sitemap with final addresses |
| Sitemap URLs not requested in the window | above 20% | Lengthen the window, then link the ignored section |
| Googlebot page requests outside the sitemap | above 30% | 410 or 301 for dead URLs, add live ones to the sitemap |
| One path with many parameter URLs | above 1,000 unique URLs | Decide which parameters change content, disallow the rest |
| Useful share of one crawler's requests | under 50% | Find what the rest is: redirects, errors, parameters |
| 404s for user-triggered AI fetchers | any | 301, content plan, or leave as is |
/.env or /.git/ answered with 200 | at least one | Today: block dotfiles, rotate every key in the file |
On this site the security check came out clean: 15,828 requests for .env files, .git folders and WordPress logins from 161 IP addresses, and zero of them got a 200. Read the report in that order too: secret files first, then the fake Googlebot share, then everything else. Two false positives to know about. On Next.js sites, scanners put the site's own address into the Referer, so junk 404s look like broken internal links. And some apps answer any unknown path, /.env included, with the normal HTML page and status 200; compare the response size with the homepage before you panic. The full checklist has 12 signals, the reason behind each threshold and how to confirm a fix.
Check your own site in 15 minutes
- Get the access logs. Your hosting panel or CDN has them. Thirty days is enough; copy them if the host keeps only a few days.
- Sort every request into a class: browser-like, scripts, SEO tools, search engines, AI bots by type.
- Verify by IP for Google, Bing, OpenAI, Perplexity, Apple and Anthropic against their published ranges.
- Open Crawl stats in Search Console and look at the split between refresh and discovery.
- Filter the 404s from ChatGPT-User and other assistants. That is a list of pages people already ask about.
The files below do steps 2 and 3 for you. The log audit kit is bash and SQL for clickhouse-local: it reads nginx or Apache logs in combined format on your machine, checks bots against 13 official lists and writes a text report with 19 sections. On its synthetic sample it goes from an empty folder to a report in about 4 seconds; ClickHouse itself is a download of about 1.3 GB, and on Windows it runs inside WSL. Nothing is uploaded. If you would rather have an AI agent do it, the prompt walks Claude Code or Codex through eight steps read-only: six questions to you, the install, the run on a copy of your logs, a findings table and a list of changes it does not apply. It shows you aggregates only. Never paste raw log lines into a chat: they contain your visitors' IP addresses. The IP range list has all 18 official files with a small script that checks a single address, and the 27-site dataset includes the scanning script if you want to repeat the measurement on your own list.
Who this method does not fit
- Sites without access logs. Vercel, Cloudflare Workers and similar platforms give you nothing to read.
- Sites where a CDN serves HTML from cache. Bots that hit the cache never reach the log, so the origin sees a fraction of the picture. Use the CDN's own logs.
- Anyone who needs live alerts. This is a batch report over a log window, not monitoring.
- Logs in other formats. JSON logs and CDN exports need their own parsing pattern first.
What I am changing after this audit
Three things. I will help Google reach the provider pages it never requested, through internal links from pages it already visits. I will create the pages ChatGPT keeps asking for, where the topic fits the directory. And I will verify bots by IP before I believe any chart about bot traffic. Then I run the same audit again ten days later and compare the same sections.
The limits of these numbers: one young site, one month, Meta and Amazon counted by user agent only, the overload example from one client site. I have seen the same Googlebot pattern, mostly refresh and little discovery, on sites in several languages, so it is not a quirk of this one. But your log will tell your own story, and reading it is a lot cheaper than guessing.
