Topic · 4 resources
Scripts, SQL and checklists for reading raw access logs: which bots really visit your site, where crawl budget goes and which requests fail.
Where search and AI crawlers publish their IP ranges: 18 JSON lists with entry counts and dates, Amazon's three pages, and a script that checks an IP address.
What you get
10 operators publish their ranges
18 JSON lists, 3,084 entries
3 Amazon pages instead of files
A prompt for Claude Code or Codex: installs clickhouse-local, audits your access logs on your own machine, proposes fixes without applying them.
What you get
8 steps the agent walks through
6 questions it asks you first
≈4 s report on the sample log
Scripts and SQL: audit nginx or Apache logs with clickhouse-local, check each bot's IP against official ranges, and get a 19-section report.
What you get
19 sections in the text report
13 official IP lists checked, 2,585 prefixes
≈4 s from zero to a report on the sample log
A checklist for reading a server log audit: 12 signals, the threshold and the reason for each, and the next step. Rules of thumb from our audits.
What you get
12 signals worth acting on
6 with a numeric threshold
6 matter at the first row
If you would rather have it done for you, send a request and describe the site.
