Discuss project

Log File Analysis for SEO: Do AI Bots Crawl More Than Google?

30 days of server logs: verified OpenAI and Perplexity bots made 4.2 times more requests than Googlebot, and 23% of 'Googlebot' was fake. Method and thresholds.

Vladislav Krivorutško — founder of ADLAB
Vladislav Krivorutško

November 9, 2026 · 23 min read

From the ADLAB channel on YouTube. Watch on YouTube

Contents
  1. 01Short answer
  2. 02Why a log file analysis shows what other SEO tools miss
  3. 03Who was in the log
  4. 04Why you cannot count Googlebot by user agent
  5. 05What Googlebot did: the log against Crawl stats
  6. 06What AI bots fetched
  7. 07What ChatGPT opens, and the pages it asks for that do not exist
  8. 08When AI bots overload a server
  9. 09robots.txt for AI bots: close training, keep search
  10. 10Throttling AI bots in nginx instead of blocking
  11. 11Server log thresholds: when to act
  12. 12Check your own site in 15 minutes
  13. 13Who this method does not fit
  14. 14What I am changing after this audit
  15. Check yourself
  16. Files from the video

TL;DR - key points

  1. 1On one of my sites, 30 days and 141,020 requests: verified OpenAI and Perplexity bots made 7,356 requests, verified Google bots 1,747, which is 4.2 times fewer
  2. 2531 of the 2,278 requests that called themselves Google (23%) came from 21 IP addresses outside Google's published ranges: count bots by IP, never by user agent
  3. 3Googlebot never requested 74% of the sitemap in 30 days, and Crawl stats in Search Console shows 98% refresh and 2% discovery: Google re-checks known pages instead of exploring
  4. 4Blocking every AI bot has a price: OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. Close training bots, slow them down, let search bots in
  5. 5A first check takes about 15 minutes: access logs, classes, IP verification, Crawl stats, and the 404s that assistants hit

Short answer

On one of my own sites, AI bots made more than four times as many requests as Google. I took 30 days of server logs, 141,020 requests, and checked every bot against the IP ranges its operator publishes. Verified OpenAI and Perplexity bots made 7,356 requests. Verified Google bots made 1,747. With Anthropic's verified ClaudeBot added, the ratio is about 5.1.

Two more findings matter more than the ratio. A quarter of the requests that called themselves Googlebot were fake. And ChatGPT kept asking for pages that do not exist on the site. This is one young site of about 2,400 pages, so read it as a snapshot, not a verdict. The method below works on any site that writes access logs. The video above walks through the same audit, and the files for repeating it on your own logs are below the article.

Why a log file analysis shows what other SEO tools miss

A site crawler shows what exists on your site. Search Console shows what Google decided to crawl and index, and only for Google. The access log is the one record of what every bot actually requested and what your server answered: status code, bytes, user agent, IP address.

The test site is a price directory for local services: about 2,400 pages on Next.js behind Cloudflare. Cloudflare does not cache its HTML, so every page request reaches the origin server and lands in the log. That detail decides whether a log analysis is possible at all. If your CDN serves HTML from cache, those requests never reach your server. On Cloudflare, cf-cache-status: DYNAMIC on the homepage means the request went to the origin. Hosts that write no access logs, such as Vercel or Cloudflare Workers, leave nothing to analyse.

The audit ran locally with ClickHouse: 31 log files, 3.6 MB, 30.6 days from 8 October to 8 November 2026, every line parsed. Claude Code ran the steps and read the report, but the counting is plain SQL. I want numbers I can reproduce, not numbers a model produced.

Who was in the log

Less than half of the traffic looks like browsers, and that is not the same as people. On 2 November alone there were 12,743 browser-like requests.

ClassRequestsShare
Browser-like clients61,52343.6%
Scripts27,35119.4%
SEO tools (Ahrefs 4,671, Semrush 3,011, DataForSEO 2,256)11,9378.5%
Facebook link-preview bot9,5536.8%
AI search bots7,3035.2%
Empty user agent4,8123.4%
AI training bots4,7803.4%
My own server checks4,4433.2%
Bing2,9872.1%
AI user-triggered fetchers2,3141.6%
Google, all requests that claim it2,2781.6%
Apple1,3611.0%

All AI classes together come to 14,397 requests, 10.2% of the log. Google is 1.6%, and that figure still includes the fakes. Facebook's link-preview bot alone made more than five times as many requests as verified Google.

Why you cannot count Googlebot by user agent

A user agent is a text header. Any script can write "Googlebot" in it. Of the 2,278 requests that called themselves Google, 531 (23%) came from 21 IP addresses outside Google's published ranges. A count by user agent would have reported 2,278 Google requests and missed the split.

Real or fake GooglebotUser agent says “Googlebot”2,278 requestsIs the IP in Google’spublished ranges?✓ Yes1,747real Google✗ No531 (23%)21 IPs: fakeShare of the 2,278 requests✓ 1,747✗ 531 · 23%
How the audit sorted requests that called themselves Google: the IP address, not the user agent, decides. Same log, 8 October to 8 November 2026.

A second group was worse: 14 IP addresses kept switching names between ChatGPT-User, Claude-User, YouBot and Googlebot. They made 4,611 requests, and 73% ended in errors. Those are scanners wearing bot costumes. My rule since then: if a bot cannot be verified by IP, I do not count it as that bot. Google's own documentation tells every site, small or large, to verify Googlebot this way.

Who publishes the ranges, as of 8 November 2026:

OperatorWhat is publishedEffect on the audit
Google, Bing, Apple, OpenAI, PerplexityJSON range files, per bot for OpenAI and PerplexityVerified
AnthropicOne list for ClaudeBot, Claude-SearchBot and Claude-User, since 20 October 2026Verified
Common Crawl, DuckDuckGo, MistralJSON listsPublished, not loaded by my kit yet
AmazonThree lists on its developer pages, not as filesCounted by user agent only
Meta, ByteDanceNo listCounted by user agent only

In total, 10 operators publish 18 JSON lists with 3,084 entries. Google-Extended and Applebot-Extended have no IP ranges at all: they are robots.txt tokens, not crawlers. Two traps came up while I built the check. Six ranges sit in both OpenAI's GPTBot and OAI-SearchBot lists, so a lookup that stops at the first match marks real GPTBot requests as fake. And behind a CDN or load balancer the log must hold the real client IP (real_ip_header in nginx, mod_remoteip in Apache), otherwise every bot carries the proxy's address and looks fake.

What Googlebot did: the log against Crawl stats

Verified Googlebot made 1,708 requests in 30 days, about 56 a day, across 630 pages. The sitemap lists 2,420 URLs. Googlebot requested 623 of them, and 1,797 (74%) got no request at all. Among the provider pages, the largest section, 77% were never requested. About one Googlebot request in five (359) was for robots.txt.

Search Console tells the same story from Google's side. In Settings, Crawl stats for the last 90 days shows 98% of requests as "refresh", re-checks of pages Google already knows, and 2% as "discovery". HTML is 84% of the requests, Googlebot Smartphone made 84%, the average response time is 615 ms. Host status says the host "had problems in the past", while my log shows zero 5xx responses in the last 30 days. So that is history, but worth knowing about.

Crawl stats covers 90 days and the log covers 30, so I compare the structure, not the numbers. Both say the same thing: Google is not exploring this site, it is maintaining what it already has. My reading is that it is a young site and Google is careful with it. That is my interpretation, not something Google has said.

Is this a crawl budget problem? For a site of 2,400 pages, by Google's own scale, almost certainly not. I covered the criteria in when crawl budget actually matters. Here the 74% is a reason to check internal linking and the sitemap, not a sign of a starving site. "Not requested in the window" also does not mean "not indexed".

What AI bots fetched

There are three kinds of AI bot, and the difference decides what you should do with each. Search bots build the index behind AI answers. Training bots collect data for models. User-triggered fetchers open one page because a person asked an assistant a question.

BotClassRequestsVerified by IPNote
OAI-SearchBotsearch4,6584,31891% of requests to real sitemap pages
PerplexityBotsearch1,9451,6871,536 on a single day, 25 October
Amazonbottraining1,764not checkedAmazon's lists are not loaded
ClaudeBottraining1,7531,534 (87%)479 of its requests were for the sitemap
ChatGPT-Useruser-triggered1,5211,257Opens pages people ask about
GPTBottraining3249462% of requests ended in errors
Claude-Useruser-triggered29917Mostly scanners using the name
Verified bot requests in 30 daysRequests verified by IP, 30 daysOAI-SearchBot · search4,318Google · all verified bots1,747PerplexityBot · search1,687ClaudeBot · training1,534ChatGPT-User · user-triggered1,257GPTBot · training94OpenAI + Perplexity: 7,3564.2 times more than Google (1,747)
Requests checked against each operator's published IP ranges on one site of about 2,400 pages, 8 October to 8 November 2026. OpenAI's three bots and PerplexityBot add up to 7,356; the Google bar counts all verified Google crawlers.

OAI-SearchBot is the busiest: 3,658 requests to 143 service pages, about 26 per page in 30 days, so it re-reads each service page almost daily. PerplexityBot is not a polite guest: four fifths of its month came in one day. ClaudeBot fetched the same sitemap file 479 times. On this site the AI search bots look at the content more often than Googlebot does. Whether that turns into visibility in AI answers is a separate question, which I discuss in AEO vs GEO.

What ChatGPT opens, and the pages it asks for that do not exist

ChatGPT-User is the most useful bot in the log, because each request means a real person asked ChatGPT something that needed this site. After the homepage (685 requests), it opened specific service pages: windshield replacement and car painting 12 times each, home loans 11, tyre change 10. It comes for pages, not for guide files: in a month of logs from 839 sites, verified AI bots, ChatGPT-User included, requested llms.txt exactly once.

It also requested eight URLs that do not exist on the site: bathroom renovation, roof construction, waterproofing, gutter cleaning, attic insulation, insulation, joint renewal, and a tyre change page at the wrong address. ChatGPT guessed those addresses. For me that is a list of topics people ask about and for which I have no page yet. My rule for such 404s: if a page on that topic exists under another address, add a 301 to it. If the topic fits the site and there is no page, put it in the content plan. Otherwise leave the 404, and never redirect them all to the homepage. The difference between a 404, a 410 and a redirect is in what to do with deleted pages.

ChatGPT-User also requested my redirect path 221 times, although robots.txt disallows it. OpenAI says user-triggered fetches may ignore robots.txt. That is documented behaviour, not a bug.

What comes back is small. These are the visits in the same 30 days that arrived with an external referrer and landed on a page that answered without an error:

SourceVisitsDistinct IPs
Google12025
Bing8567
ChatGPT4120
DuckDuckGo3023

By distinct visitors Bing sent more people than Google here. Bots read thousands of pages, people arrive in dozens.

When AI bots overload a server

On a small site, load is not an issue. On one multilingual client site it was. Over 11 days a single crawler that called itself Meta's made 41,289 requests and pulled 2.4 GB, 13 times more requests than Googlebot's 3,055 on the same site. Meta publishes no IP list, so I can only say what the crawler claimed to be. About half of the AI training requests there went to redirects: waste for the bots and for the server.

The reflex of most developers I meet is to block the bots or throttle them hard. I understand it, but a bot you block cannot read your page, and an assistant that cannot read your page cannot cite it. OpenAI says it plainly: sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot may reduce visibility in its search results.

So I do it in this order:

  1. Verify by IP and drop the fakes. Part of the "AI load" is usually scanners using bot names.
  2. Separate the classes. Training bots can be slowed down or closed, search bots and user-triggered fetchers should get in.
  3. Fix the waste: redirects from your own links, parameter traps, heavy pages.
  4. Rate-limit instead of blocking. A bot that waits one second between requests makes at most 86,400 requests a day, whatever the size of the site.

The order matters: blocking first cuts off the search bots along with the crawler that caused the load. Untangling bot load this way is a regular part of my technical SEO audits.

To opt out of AI training and stay in AI search answers, disallow the training bots and the control tokens, and leave the search bots alone:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Google-Extended
Disallow: /

Stacked User-agent lines share the rule under them. OAI-SearchBot, Claude-SearchBot, PerplexityBot and everything else not named keep following your User-agent: * group. Three details catch people out:

  • Google-Extended is not a crawler. Google says it does not affect inclusion or ranking in Google Search, but it does cover grounding in Gemini Apps. Delete that line if you want to stay citable there.
  • A bot with its own group ignores the * group. If you close one section for GPTBot only, repeat your shared rules (/admin/, /cart/) inside its group, or GPTBot gets access to them.
  • User-triggered fetchers may ignore robots.txt. OpenAI, Perplexity, Meta, Amazon and Google each say so about their own. For those, a firewall rule or a throttle is the tool.

The general rules of the file itself are in sitemap.xml and robots.txt best practices. The robots.txt template in the files has four variants (open, close training, close everything, close by section) and a table of 30 bots with each operator's own statement.

What do large sites do? On 8 November 2026 I checked the public robots.txt of 27 large publishers, marketplaces and reference sites. 18 close at least one of 11 AI bots completely (Disallow: / for that bot), 9 close none. ClaudeBot and Bytespider are closed by 16 sites each, GPTBot and CCBot by 14, PerplexityBot by 11, ChatGPT-User by 8, OAI-SearchBot by 7, and Google-Extended by only 6. Training bots are closed more often than search bots of the same operator: all 7 sites that close OAI-SearchBot also close GPTBot, and 7 more close GPTBot alone. The sites were chosen by me, so this is not a sample of the web. My own small directory blocks none of these bots. For a site that wants to show up in AI answers, I think that is the right call.

Throttling AI bots in nginx instead of blocking

The nginx config in the files sorts user agents into three classes and gives each its own allowance per bot name. Googlebot, Bingbot and ordinary visitors match no pattern and are never counted.

ClassAllowance30 quick requests in the test
Training (ClaudeBot, GPTBot, CCBot and others)1 request per second, burst 1011 × 200, 19 × 429
Search (OAI-SearchBot, PerplexityBot and others)5 per second, burst 2021 × 200, 9 × 429
User-triggered (ChatGPT-User, Claude-User and others)5 per second, burst 2021 × 200, 9 × 429
Googlebot, Bingbot, Chrome, empty user agentnot counted30 × 200

A refused request gets 429 Too Many Requests with Retry-After: 10. I tested it in Docker on nginx 1.30.6 and 1.29.3. Run it as a dry run for a week first: nginx then only logs what it would have refused.

Three limits. No operator page I read says how its crawler reacts to a 429; Anthropic and Common Crawl document support for Crawl-delay, Amazon and Apple say their bots ignore it, Google does not support it. The throttle works on the user agent, so fakers spend the real bot's allowance; the throttle config notes how to key it by IP instead. And Googlebot stays out of the list on purpose: Google's guide on reducing crawl rate allows 429, 500 or 503 for one to two days only and warns that URLs answering like that for longer may drop from the index.

Server log thresholds: when to act

A number in the report is a reason to look, not a verdict. These are rules of thumb from three trial audits of production sites in October 2026, not Google thresholds. Google publishes no percentage for any of them.

SignalAct whenFirst step
Requests claiming Google that fail IP verificationanyUse verified hits only in every Googlebot figure
5xx for verified Googlebotabove 1% of its requestsError logs for those paths and times
Redirects for verified Googlebotabove 10%Replace links in templates, menus and the sitemap with final addresses
Sitemap URLs not requested in the windowabove 20%Lengthen the window, then link the ignored section
Googlebot page requests outside the sitemapabove 30%410 or 301 for dead URLs, add live ones to the sitemap
One path with many parameter URLsabove 1,000 unique URLsDecide which parameters change content, disallow the rest
Useful share of one crawler's requestsunder 50%Find what the rest is: redirects, errors, parameters
404s for user-triggered AI fetchersany301, content plan, or leave as is
/.env or /.git/ answered with 200at least oneToday: block dotfiles, rotate every key in the file

On this site the security check came out clean: 15,828 requests for .env files, .git folders and WordPress logins from 161 IP addresses, and zero of them got a 200. Read the report in that order too: secret files first, then the fake Googlebot share, then everything else. Two false positives to know about. On Next.js sites, scanners put the site's own address into the Referer, so junk 404s look like broken internal links. And some apps answer any unknown path, /.env included, with the normal HTML page and status 200; compare the response size with the homepage before you panic. The full checklist has 12 signals, the reason behind each threshold and how to confirm a fix.

Check your own site in 15 minutes

  1. Get the access logs. Your hosting panel or CDN has them. Thirty days is enough; copy them if the host keeps only a few days.
  2. Sort every request into a class: browser-like, scripts, SEO tools, search engines, AI bots by type.
  3. Verify by IP for Google, Bing, OpenAI, Perplexity, Apple and Anthropic against their published ranges.
  4. Open Crawl stats in Search Console and look at the split between refresh and discovery.
  5. Filter the 404s from ChatGPT-User and other assistants. That is a list of pages people already ask about.

The files below do steps 2 and 3 for you. The log audit kit is bash and SQL for clickhouse-local: it reads nginx or Apache logs in combined format on your machine, checks bots against 13 official lists and writes a text report with 19 sections. On its synthetic sample it goes from an empty folder to a report in about 4 seconds; ClickHouse itself is a download of about 1.3 GB, and on Windows it runs inside WSL. Nothing is uploaded. If you would rather have an AI agent do it, the prompt walks Claude Code or Codex through eight steps read-only: six questions to you, the install, the run on a copy of your logs, a findings table and a list of changes it does not apply. It shows you aggregates only. Never paste raw log lines into a chat: they contain your visitors' IP addresses. The IP range list has all 18 official files with a small script that checks a single address, and the 27-site dataset includes the scanning script if you want to repeat the measurement on your own list.

Who this method does not fit

  • Sites without access logs. Vercel, Cloudflare Workers and similar platforms give you nothing to read.
  • Sites where a CDN serves HTML from cache. Bots that hit the cache never reach the log, so the origin sees a fraction of the picture. Use the CDN's own logs.
  • Anyone who needs live alerts. This is a batch report over a log window, not monitoring.
  • Logs in other formats. JSON logs and CDN exports need their own parsing pattern first.

What I am changing after this audit

Three things. I will help Google reach the provider pages it never requested, through internal links from pages it already visits. I will create the pages ChatGPT keeps asking for, where the topic fits the directory. And I will verify bots by IP before I believe any chart about bot traffic. Then I run the same audit again ten days later and compare the same sections.

The limits of these numbers: one young site, one month, Meta and Amazon counted by user agent only, the overload example from one client site. I have seen the same Googlebot pattern, mostly refresh and little discovery, on sites in several languages, so it is not a quirk of this one. But your log will tell your own story, and reading it is a lot cheaper than guessing.

Quiz

Check yourself

A few questions on the article. The answer and the reasoning appear as soon as you pick; nothing is sent anywhere.

0/4

answered

  1. 1.Your log shows 2,278 requests with a Googlebot user agent. What do you need before you report them as Google's crawl?
  2. 2.You want to stay out of AI model training but keep being cited in ChatGPT search answers. What goes into robots.txt?
  3. 3.ChatGPT-User requested eight URLs that return 404 on your site. What is the right step?
  4. 4.On a 2,400-page site Googlebot did not request 74% of the sitemap URLs in 30 days. What does that most likely mean?

ADLAB · files from the video

Files from the video

Scripts, prompts, templates and data shown in the video. Text files open as plain text; the zip holds every file of one item.

  1. 01Script · SQL12 files · 484 KB · MIT

    Server log audit kit: tell real Googlebot from fakes with ClickHouse

    Scripts and SQL: audit nginx or Apache logs with clickhouse-local, check each bot's IP against official ranges, and get a 19-section report.

    What's inside · 12 files
    • README.md

      Requirements, install, 15-minute run, log format, how to read the report, limits

      8.3 KB
    • run.sh

      Runs the whole audit: logs, IP ranges, classification, sitemap, report

      8.8 KB
    • install.sh

      Installs the ClickHouse binary with ClickHouse's own installer; --dry-run checks first

      5.6 KB
    • build_combined.sql

      Parses nginx/Apache combined logs into logs.parquet

      1.6 KB
    • enrich.sql

      Adds bot name, traffic class, verified flag (IP in official ranges) and page or asset

      7.0 KB
    • report.sql

      The 19 sections of the text report

      12 KB
    • ranges.sh

      Downloads the official crawler IP ranges (Google, Bing, Apple, OpenAI, Perplexity, Anthropic)

      4.9 KB
    • sitemap.sh

      Collects all sitemap URLs (optional step)

      2.6 KB
    • sample/make_sample_log.py

      Generator for the synthetic sample log

      13 KB
    • sample/access.sample.log

      Synthetic log of 1,954 lines for example.com, with real and fake bots

      415 KB
    • sample/sitemap.sample.txt

      The 108 sitemap URLs that go with the sample log

      3.7 KB
    • LICENSE

      MIT License, ADLAB OÜ, 2026

      1.0 KB
  2. 02Dataset4 files · 22 KB · CC-BY-4.0

    How 27 big sites treat AI crawlers in robots.txt (data, 2026-11-08)

    27 large sites checked on 2026-11-08: who closes ClaudeBot, GPTBot, CCBot or PerplexityBot completely in robots.txt. CSV, script and method, CC BY 4.0.

    What's inside · 4 files
    • ai-bots-robots-27-sites.csv

      One row per site: 11 AI crawler tokens, true when the robots.txt closes the bot with Disallow: /

      4.7 KB
    • scan_robots.py

      Python 3.8+, standard library only; reproduces the measurement for any list of domains

      13 KB
    • sites.txt

      The 28 domains that were requested

      499 B
    • README.md

      Definition, CSV columns, how to re-run, limits

      3.6 KB
  3. 03Dataset · Script2 files · 6.5 KB · MIT

    Official crawler IP ranges: 18 lists and an IP checker

    Where search and AI crawlers publish their IP ranges: 18 JSON lists with entry counts and dates, Amazon's three pages, and a script that checks an IP address.

    What's inside · 2 files
    • crawler-ip-ranges.csv

      21 rows: operator, bots, class, list address, format, entry counts, the list's own date, check date, operator page

      3.9 KB
    • check_ip.py

      Python script: says which operator's official list contains an IP address

      2.6 KB
  4. 04Prompt2 files · 8.9 KB · MIT

    Prompt: have an AI agent set up ClickHouse and audit your server logs

    A prompt for Claude Code or Codex: installs clickhouse-local, audits your access logs on your own machine, proposes fixes without applying them.

    What's inside · 2 files
    • prompt.md

      The prompt, ready to paste into an agent with a terminal

      7.2 KB
    • README.md

      How to use it and what to check before you trust it

      1.7 KB
  5. 05Template · Config2 files · 16 KB · MIT

    robots.txt for AI bots: block training, keep search

    Copy-paste robots.txt variants for AI crawlers: opt out of training, close everything, or block single sections. Reference table of 30 bots with sources.

    What's inside · 2 files
    • robots-ai-bots.txt

      Commented robots.txt template with four ready variants: open, block training, block all, block sections

      6.6 KB
    • ai-bots-reference.csv

      30 bots: operator, class, robots.txt behavior and IP-range list per the operator's own page

      9.5 KB
  6. 06Checklist1 file · 11 KB · MIT

    Server log audit: what each number means and when to act

    A checklist for reading a server log audit: 12 signals, the threshold and the reason for each, and the next step. Rules of thumb from our audits.

  7. 07Config3 files · 14 KB · MIT

    Throttle AI crawlers in nginx instead of blocking them

    Tested nginx config: 429 with Retry-After when ClaudeBot, GPTBot or CCBot pass 1 request per second. Googlebot and Bingbot are never counted.

    What's inside · 3 files
    • nginx-ai-bots-throttle.conf

      http-level include: User-Agent maps, rate-limit zones, 429 with Retry-After, rejection log

      6.1 KB
    • README.md

      Install, dry run, rollback and how to change the numbers

      1.9 KB
    • test-throttle.sh

      Functional test in a throwaway Docker container; needs bash, docker and curl

      6.0 KB

Frequently asked questions

How do I tell a real Googlebot from a fake one?
Check the IP address, not the user agent. Google publishes its crawler ranges as JSON files and also documents a reverse and forward DNS check. A request that says Googlebot but comes from an IP outside those ranges is not Google. In one 30-day audit of my own site, 531 of 2,278 requests that called themselves Google came from 21 such addresses.
Do AI bots really crawl sites more than Googlebot?
On the site I measured, yes: verified OpenAI and Perplexity bots made 7,356 requests in 30 days against 1,747 from verified Google bots. That is one young site of about 2,400 pages, so it is a snapshot, not a rule. On a large site with strong links Googlebot can easily dominate. The only way to know for your site is to count verified requests in your own access logs.
Should I block GPTBot and ClaudeBot in robots.txt?
If you object to model training, blocking these training bots is reasonable and does not, according to the operators' pages, remove you from their search answers. Keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed if you want to be cited. OpenAI states that sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, so blocking every AI bot removes that channel.
Does ChatGPT-User follow robots.txt?
Not necessarily. OpenAI says robots.txt rules may not apply to user-triggered fetches, because a person asked for that page. In my log ChatGPT-User requested a path that robots.txt disallows 221 times in 30 days. If a section must stay closed to such fetchers, that takes a server or firewall rule, not robots.txt.
How many days of logs do I need for an SEO log analysis?
Thirty days is a workable window for a site with a few thousand URLs. With five or seven days, a high share of sitemap URLs 'not requested' mostly means Googlebot has not got round to them yet. Many hosts keep only a few days of logs, so copy them weekly if you want a longer window.
Vladislav Krivorutško — founder of ADLAB

About the author

Vladislav Krivorutško

Founder of ADLAB OÜ · SEO and Google Ads

Over 20 years in search traffic and monetization, and on the Estonian market since 2017. I work solo: I run the audit, build the strategy and deliver the project myself — no subcontractors, no templates. I only write about what I have tested on my own and client sites.

  • 20+ years in search traffic
  • 50+ end-to-end projects
  • Own sites in competitive niches
  • SEO for ru/et/en in one market
More about me

Conclusion

A server log is the only place where you see what bots really request and what your server answers. On my site AI bots made several times more requests than Google, a quarter of the 'Google' traffic was fake, and ChatGPT asked for pages I do not have yet. None of that shows up in a crawler or in Search Console. Start with IP verification, then decide which bot class you close, slow down or let in. If the log shows 5xx for Googlebot, parameter traps or bots overloading the server, that is work for a technical SEO audit.

Read next