ADLAB resources
Scripts, prompts and configs from real audits
Seven resources built for our own audits and videos. Every page shows the version, the date and what it was tested on.
- resources
- 7
- topics
- 2
- licenses named on every page
- MIT · CC BY
Two topics
Start with the problem you have
Files are grouped by the task they solve, not by the video they came from.
Server logs and crawl analysis
Scripts, SQL and checklists for reading raw access logs: which bots really visit your site, where crawl budget goes and which requests fail.
- Official crawler IP ranges: 18 lists and an IP checkerdataset · script
- have an AI agent set up ClickHouse and audit your server logsprompt
- Server log audit kit: tell real Googlebot from fakes with ClickHousescript · sql
- Server log audit: what each number means and when to actchecklist
AI bots and AI search
robots.txt templates, rate limits and measurements for the crawlers behind ChatGPT, Claude, Perplexity and Google's AI features.
- How 27 big sites treat AI crawlers in robots.txt (data, 2026-11-08)dataset
- robots.txt for AI bots: block training, keep searchtemplate · config
- Throttle AI crawlers in nginx instead of blocking themconfig
From one 30-day audit of a single site. Your numbers will differ: that is the point of running it on your own logs.
START HERE
Server log audit kit: tell real Googlebot from fakes with ClickHouse
Scripts and SQL: audit nginx or Apache logs with clickhouse-local, check each bot's IP against official ranges, and get a 19-section report.
- 19 sections in the text report
- 13 official IP lists checked, 2,585 prefixes
- ≈4 s from zero to a report on the sample log
- 0 bytes of your logs uploaded anywhere
All resources
Server logs and crawl analysis
4 resourcesOfficial crawler IP ranges: 18 lists and an IP checker
Where search and AI crawlers publish their IP ranges: 18 JSON lists with entry counts and dates, Amazon's three pages, and a script that checks an IP address.
Prompt: have an AI agent set up ClickHouse and audit your server logs
A prompt for Claude Code or Codex: installs clickhouse-local, audits your access logs on your own machine, proposes fixes without applying them.
Server log audit kit: tell real Googlebot from fakes with ClickHouse
Scripts and SQL: audit nginx or Apache logs with clickhouse-local, check each bot's IP against official ranges, and get a 19-section report.
Server log audit: what each number means and when to act
A checklist for reading a server log audit: 12 signals, the threshold and the reason for each, and the next step. Rules of thumb from our audits.
AI bots and AI search
3 resourcesHow 27 big sites treat AI crawlers in robots.txt (data, 2026-11-08)
27 large sites checked on 2026-11-08: who closes ClaudeBot, GPTBot, CCBot or PerplexityBot completely in robots.txt. CSV, script and method, CC BY 4.0.
robots.txt for AI bots: block training, keep search
Copy-paste robots.txt variants for AI crawlers: opt out of training, close everything, or block single sections. Reference table of 30 bots with sources.
Throttle AI crawlers in nginx instead of blocking them
Tested nginx config: 429 with Retry-After when ClaudeBot, GPTBot or CCBot pass 1 request per second. Googlebot and Bingbot are never counted.
Before you download
Runs on your machine
Everything runs locally, so read a script before you run it. Server logs contain visitor IP addresses: keep them private and do not paste raw logs into a chat with an AI model.
Licenses on every page
Scripts and templates are MIT-licensed, datasets are CC BY 4.0, and each page names its license. A copy you download is yours to use and change.
Counted without cookies
Downloads are counted on our server without cookies, and the collector stores a hash of the IP address, not the address. We use the totals to decide what to maintain.
Want an audit like this on your site?
If you would rather have it done for you, send a request and describe the site.