ADLAB logo
  • checklist

Server log audit: what each number means and when to act

A checklist for reading a server log audit: 12 signals, the threshold and the reason for each, and the next step. Rules of thumb from our audits.

  • v1.0
  • updated 8 November 2026
  • MIT
  • Vladislav Krivorutško
Download log-audit-thresholds.mdJump to the text

The log audit report has 12 signals worth acting on: six with a numeric threshold and six that matter as soon as one row appears. Every threshold below is a rule of thumb from our audits, not an official Google threshold. We found no percentage published by Google for any of them.

The numbers come from three trial audits of production sites in October 2026, run with the report of the log-audit-kit resource (/en/resources/logs/log-audit-kit). On 2026-11-08 we re-read them against Google's crawl budget guide. A crossed line is a reason to open the rows behind it, not a verdict.

Files in this resource

1 file

  • log-audit-thresholds.md

    Printable cheat sheet: 12 signals, thresholds, reasons and next steps

    11 KB

The 12 signals

#Signal (report section)ThresholdWhy this number
1Fake Googlebot share (3)Any. Count only verified hitsThe User-Agent is free text. In one of our audits, 531 of 2,278 requests that called themselves Google (23 %) came from 21 IPs outside Google's ranges. We set no percentage because the action is the same at 2 % and at 40 %: every Googlebot figure uses verified hits only.
25xx for verified Googlebot (4, 5, 14)More than 1 % of its requestsGoogle's crawl budget guide says server errors and HTTP 429 lower the crawl capacity limit, so Google crawls less. It gives no percentage. We use 1 %: above one error per 100 requests the errors form a pattern, and the cause (a slow page, a crashing plugin, rate limiting) deserves a look the same day.
3Redirects for verified Googlebot (4, 5)More than 10 % of its requestsA redirect is a request that returns no content, and the guide advises against long redirect chains. Redirects from old backlinks are normal and not yours to fix. Redirects from your own links and sitemap are yours. Past 10 % we check which kind dominates.
4Sitemap URLs Googlebot did not request in the window (6)More than 20 %The sitemap lists what you want crawled. A fifth without a single request can mean three things: the log window is too short for the site, Google does not rank those URLs as worth a visit, or they are hard to reach. Section 6 splits the share by first path segment, so you see which part is ignored. Not requested in the window is not the same as not indexed.
5Googlebot requests outside the sitemap (7)More than 30 % of its page requestsGooglebot finds URLs through links as well as the sitemap, so some share is normal. A large one means its requests go to addresses you do not list: old URLs, wp-json, feeds, parameter URLs. Section 7 lists the top rows.
6Parameter URLs on one path (9)More than 1,000 unique URLs on one pathFilters, sort orders and session ids multiply into an open-ended set of URLs, and crawlers follow them. The guide advises blocking unimportant variants such as sorted versions of a page in robots.txt. We use 1,000 because on a site with a few hundred real pages, one path with 1,000 variants already outnumbers the site.
7Useful share of a crawler's requests (8, 10)Under 50 %"Useful" means status 200, URL in the sitemap, no parameters. Under half means the rest goes to redirects, errors, parameter variants or pages missing from the sitemap. AI bots may reach pages through links rather than your sitemap, so read "outside the sitemap" for them with care.
8Pages opened by ChatGPT-User (11)No threshold, read the listChatGPT-User fetches a page when a person's conversation needs it, and OpenAI says robots.txt rules may not apply to it. The list is the set of pages an assistant actually read to answer a question.
9404s for user-triggered AI bots, the ai-user class (11)AnyAssistants sometimes request URLs a model invented. The report drops scanners and secret-file probes and lists what is left. Each row is a page the assistant expected to find.
10404s with an internal Referer (13)AnyThe Referer is your own site, so a page of yours links to an address that does not exist. These are the cheapest fixes in the report.
11404s with an external Referer (13)AnyAnother site links to an address that no longer exists, so that link passes nothing to you.
12/.env or /.git/* answered with 200 (17)At least oneThese files hold passwords, keys and source history. Public sites are probed for them constantly, and most probes get 404. A 200 means a file, or something that looks like one, was served.

Who can skip which rows

Rows 4, 5 and 7 measure how Googlebot spends its requests. Google's crawl budget guide says a site whose new pages get crawled the same day can skip that guide, and it is written for sites with 1 million or more pages that change weekly, or 10,000 or more that change daily. On a smaller site, read rows 4, 5 and 7 as housekeeping, not as lost rankings. Rows 1, 2, 9, 10, 11 and 12 apply to every site: they are about fake bots, server errors, broken links and exposed files. If your host writes no access logs (Vercel and Cloudflare Workers, for example), there is nothing to measure.

What to do next

After any fix, rerun the kit on logs written after the change, 7 to 10 days later, and compare the same section.

#SignalDo thisCheck that it worked
1Fake GooglebotUse only verified rows for every Googlebot figure (the report does). If the fakes hit expensive pages or login URLs, rate-limit or block their IPs at the firewall or CDN. Do not block by User-Agent: the real Googlebot sends it too.Section 3: spoofed_hits falls, verified_hits stays.
25xxOpen the web server and application error logs for the paths in sections 5 and 14, at the times of the errors. Fix the cause: a slow query, a crashing plugin, a resource limit, a rate limit that catches Googlebot.Section 4: pct_5xx under 1 %.
3RedirectsTake the top URLs of section 5. Replace links to them in templates, menus and the sitemap with the final addresses. Keep the 301s for old backlinks. Cut chains to one hop.Section 4: pct_redirect under 10 %.
4Sitemap not requestedLengthen the log window first. If the share stays, link the ignored section from pages Googlebot already visits, check that those URLs are indexable, and keep lastmod in the sitemap true.Section 6: pct_not_seen falls for that segment.
5Outside the sitemapFor each top row of section 7: a dead URL gets a 410 or a 301 to its replacement, a useful page missing from the sitemap gets added, and junk patterns (feeds, wp-json, internal search) get a robots.txt disallow.Section 7: pct_of_page_hits under 30 %.
6Parameter trapDecide which parameters change the content. Disallow the rest in robots.txt (sort, session, tracking), stop linking to filter combinations nobody searches for, and remove them from the sitemap. A noindex tag does not stop the requests.Section 9: uniq_urls for that path drops.
7Low useful shareRead the same row for what the rest is made of: pct_params, pct_redirect, pct_4xx, pct_200_outside_sm. Fix the biggest one with the actions above. For an AI bot you do not want, disallow its group in robots.txt.The row's pct_useful rises above 50 %.
8ChatGPT-User pagesKeep those pages accurate and dated, put the key facts in the HTML the server sends, and link to them from other pages. Assistants open them to answer questions, so they come first.They stay in the list, with no 404 mixed in.
9Assistant 404sFor each URL: if a page on that topic exists under another address, add a 301 to it. If the topic fits the site and no page exists, put it in the content plan. Otherwise leave the 404. Do not redirect them all to the homepage.Section 11: the URL is gone from the 404 table.
10Internal 404sOpen the referring page, find the link and correct it. If many pages use it, add a 301 instead. First confirm the link exists in the HTML (see the Next.js trap below).Section 13: the internal rows are gone.
11External 404sOpen the referring page and check that it is real and links to you. If it does, 301 the dead URL to the closest live page.Section 13: the external rows are gone.
12Secret files with 200Today: block dotfiles at the web server, remove the files from the web root, change every password and key that was in the file, and check whether /.git/config or /.git/HEAD was served.Request the URL from outside: it answers 403 or 404. Section 17 is empty on the next run.

Rather have this done for you?

Send the site address and what you want to know. We run the audit and send the report.

Send a request

Read the report in this order

  1. Section 17. A 200 on /.env or /.git/* goes first, before anything else.
  2. Section 3. It tells you how much of the "Google" traffic is real, so the rest of the numbers can be trusted.
  3. Sections 4, 5 and 14: errors and redirects for verified Googlebot.
  4. Sections 6, 7, 8 and 9: where crawling goes compared with the sitemap.
  5. Sections 10 to 13: AI bots, assistants and 404s with a Referer.

This order is our suggestion, not part of any official guide.

False positives and traps

  • Counting bots by User-Agent. The header can be set by anyone. Use the verified column.
  • Next.js sites. Scanners put the site's own address into the Referer (/wp, /graphql, /wp-json). The 404s then show up as "internal Referer", but no page of yours links there. We saw this on Next.js; other frameworks may behave the same. Search the HTML of your pages for the URL before fixing anything.
  • A 200 on /.env that is only your app shell. Some sites answer 200 with the normal HTML for any unknown path. Compare min_bytes and max_bytes in section 17 with the size of your homepage, and check status and size from outside without printing the body: curl -s -o /dev/null -w '%{http_code} %{size_download}' https://example.com/.env.
  • Forged Referer. The header comes from the client. Before redirecting an external 404, open the referring page.
  • Short windows. Many hosts keep a few days of logs. On a site with thousands of URLs, a 5-day window shows a large "not requested" share only because Googlebot has not got round to them yet. Google calls the page counts in its crawl budget guide a rough estimate.
  • Young sites. Our opinion, not a Google statement: Google visits a site that is a few months old cautiously, so a high "not requested" share there is common and not by itself a fault.
  • Vendor lists change. When we first ran these audits, Anthropic published no IP ranges. On 2026-11-08 it publishes them, and kit 1.0 uses them. Meta's crawler page lists none, so Meta bots stay unverified.

Files are provided as is: run them on your own data and check the result.

More files from the same audit

Want an audit like this on your site?

If you would rather have it done for you, send a request and describe the site.

Send a request
Download log-audit-thresholds.md