Short answer
Crawl budget is how many of your URLs Googlebot fetches in a given period. It rarely becomes the bottleneck: on sites under a few thousand addresses, unindexed pages almost always have another cause — thin content, duplicates, or no internal links. Crawl budget deserves attention once a site generates tens of thousands of URLs: a filtered catalogue, a booking calendar, on-site search, sorting parameters.
What crawl budget is made of
Google Search Central describes it as the interaction of two constraints.
| Component | What it is | Who controls it |
|---|---|---|
| Crawl rate limit | How many requests the site takes without degrading | You — through server response time and 5xx errors |
| Crawl demand | How much Google wants those URLs | Google — via page popularity, update frequency, staleness |
Which leads to the main point, and Google Search Central says it outright: you cannot increase the budget directly. You can make the site respond faster and make Googlebot spend its requests on useful addresses instead of junk. Everything else is folklore.
When crawl budget is not the culprit
People regularly come to me saying "Google is not indexing the site, probably not enough crawl budget". Usually the site has a couple of hundred pages and crawl budget has nothing to do with it.
Signs the cause is elsewhere:
- Crawl stats show Googlebot visiting regularly, yet pages sit in "Crawled — currently not indexed" — that is a question of content value, not crawl frequency;
- the unindexed pages have no internal links at all — then they are orphan pages and there is nowhere to crawl them from;
- near-identical product pages fail to enter the index — then the work is on duplicate content, not on the budget.
I collected the full list of causes in a separate article on why Google is not indexing your pages. Crawl budget is the last item there, and that is the correct order to check things.
What actually eats the budget
Almost always it is not the content but the addresses the site generates on its own, and Googlebot has no way to tell them apart from the useful ones.
- Faceted navigation. Filters that combine with each other explode combinatorially: three filters with ten values each is already a thousand addresses per category.
- Sorting and parameterised pagination. The same product is reachable through a dozen different URLs.
- On-site search. Every query becomes its own results page unless you block it.
- UTM tags on internal links. A classic: tags added to in-site banners, and every page doubles.
- Booking calendars. Endless generation of future dates is a trap the crawler never leaves.
- Redirect chains. Each hop is a separate request: three instead of one costs three times the budget.
- Soft 404s. Deleted products still answering 200 with an empty page keep getting crawled. How to close such addresses properly is covered in my article on 404, 410 or a redirect for deleted pages.
How to diagnose it in an hour
The order I follow on a technical audit:
- Search Console → Settings → Crawl stats. Three things: requests per day, average response time, and the breakdown by purpose (discovery vs refresh) and by file type.
- Breakdown by response code. A noticeable share of 3xx and 4xx means budget is going to addresses that no longer exist.
- Server logs. The only source that shows, URL by URL, what Googlebot fetched and how often. I pull 2–4 weeks, filter by user agent with a reverse DNS check, and group by URL pattern. Usually a handful of patterns explain most of the crawling.
- Your own crawl. Compare how many URLs a crawler finds through links with how many exist in the sitemap. A multiple-fold gap is a reliable sign of junk generation.
- Response time. If average response time sits around a second or higher, crawl frequency stays low no matter what else you do. This is adjacent to site speed and rankings, but here what matters is server response, not Core Web Vitals.
A one-off crawl spike after a release is normal. The warning sign is a sustained shift toward parameterised URLs while crawls of product pages fall.
The tools that actually control it
| Tool | What it does | When to use it |
|---|---|---|
| robots.txt | Prevents the URL from being requested | On-site search, calendars, obviously junk parameters |
| noindex | Removes from the index, crawling continues | The page is useful to visitors but not to search |
| rel=canonical | Consolidates duplicates | Sorting variants and product variations |
| Parameter-free links | Creates no extra addresses | Internal banners: keep tags for external sources |
| Fast server response | Raises crawl frequency | Always |
| Accurate lastmod in the sitemap | Helps Google pick what to recrawl | Catalogues that change often |
Sitemap and robots file setup is covered separately in sitemap.xml and robots.txt best practices. One caveat belongs here: robots.txt is not a way to remove a page from search results. A URL blocked there can still be shown without a description, precisely because Google forbade itself from reading it.
What I see on my own projects
On my own sites Googlebot hit the crawl ceiling exactly once, and not because of size but because of architecture: comparison pages generated from parameter combinations produced an order of magnitude more addresses than there was useful content. The fix was not crawl tuning but a decision about which combinations deserve to exist as a page at all.
An honest limit of applicability: if you run a 40-page service site or a 300-article blog, this whole chapter is not for you. Your ceiling is demand and quality, not crawling. The topic becomes relevant when your URL count starts being measured in tens of thousands and the share of crawls going to important pages drops.
What to do this week
- Open Crawl stats and write down three numbers: requests per day, average response time, share of non-200 responses.
- Crawl the site and compare the URL count found through links with the sitemap.
- Find the three largest patterns of junk addresses and decide for each: robots.txt, canonical, or removing the links.
- Strip UTM tags from internal links — the cheapest fix with a visible effect.
- If server response is slower than a second, work on the backend: it affects both crawling and users. Those fixes belong to web development rather than SEO settings, but without them a technical audit hits a ceiling.
