Short answer
If robots.txt returns a 404, Google decides the file does not exist and there are no restrictions. All your Disallow rules stop working at once. If it returns a 5xx, Google stops crawling the site for 12 hours, then lives for up to 30 days on the last good copy of the file. Googlebot does follow a redirect, at least five hops in a row, and then treats the result as a 404.
That leads to a non-obvious conclusion. The danger is not so much the file being unreachable as the rules quietly disappearing: a 404 removes them instantly, a long-lasting error removes them after 30 days. The site keeps working throughout, and nobody notices.
What Google does with each response
The rules below come from Google's documentation on how it interprets robots.txt. I opened it on 9 November 2026 and checked every row of the table against it. What I measured myself is marked in the next section.
| Response to /robots.txt | What Google does | What it means for the site |
|---|---|---|
| 2xx | Processes the file as served, caches it, generally for up to 24 hours | Works as intended |
| 3xx | Follows at least five hops, then treats the result as a 404 | One hop (http to https) is safe; a long chain equals a missing file |
| 4xx, except 429 | Treats it as if no valid file exists | All Disallow rules stop applying |
| 5xx, first 12 hours | Stops crawling the site while retrying the file | New pages are not crawled in that window |
| 5xx, up to 30 days | Uses the last good copy and keeps trying for a new one | Rules hold, if a copy existed |
| 5xx beyond 30 days | For a generally available site, treats it as having no file | Rules are dropped |
The documentation page says nothing specific about 429; it only excludes it from the 4xx rule. It does not say what Google does with it, so I file 429 under server errors as a working assumption, not as a fact.
Three more limits. Google does not read a file larger than 500 KiB in full: everything past that mark is ignored. The cache usually lives up to 24 hours and can last longer when a refresh fails. And rules apply only to the host, protocol and port where the file sits: https://adlab.ee and https://www.adlab.ee are two different files.
What this looks like on adlab.ee: a measurement from 9 November 2026
I ran curl -I against six URL variants. The file https://adlab.ee/robots.txt returns 200 with text/plain, weighs 77 bytes and has three meaningful lines: Allow: /, Disallow: /dl/ and a link to the sitemap. The header Cache-Control: public, max-age=14400 lets Cloudflare and browsers keep a copy for 4 hours.
Four of the other five addresses (both http ones, https://www and the one with a trailing slash) return 308 and redirect once to https://adlab.ee/robots.txt. Google's limit of five hops leaves a margin of four. The fifth, /Robots.txt with a capital R, returns 404, so keep the file name lowercase.
What I did not measure: 5xx and 429. I will not take robots.txt down on a live site for an experiment, so the table rows about 12 hours and 30 days rest on Google's documentation, not on my own observations.
Where 5xx and 404 actually come from
In my practice the file rarely breaks on its own; it breaks together with something else: a release, a plugin, a migration. Five scenarios, each ending in 403, 404, 429 or 5xx:
- Maintenance mode returning 503 for the whole site. Google recommends 503 for short stoppages, and for pages that is right. But if the maintenance page also covers
/robots.txt, the 12-hour rule applies: crawling stops. Exclude the file from the maintenance page. - Bot protection and rate limiting. A CDN or firewall answers 429 or 403 to requests without browser headers. To Google a 403 equals a 404, so the file is "gone" together with all its rules.
- A virtual file in the CMS. WordPress serves
robots.txtdynamically when no physical file exists. A broken plugin turns it into a 500 while the other pages keep working from cache. - A release from staging. The test environment's
Disallow: /lands on the live site. The file returns 200 and this is the most expensive case; I covered it in the article on staging and production. - A redirect chain on the file itself. A domain migration added a hop, then another, and the final file is more than five hops away. How to find such chains is covered in the piece on redirect chains.
If a page blocked with Disallow stays in Google's results instead of disappearing, the reason is that blocking crawling is not the same as forbidding indexing (noindex). I described that difference in the article on sitemap.xml and robots.txt.
How to check in a minute
Run the four URL variants once and make sure each has one hop and ends with a 200:
for u in http://site.ee https://site.ee http://www.site.ee https://www.site.ee; do
echo "== $u"; curl -sI -m 15 "$u/robots.txt" | grep -iE "^(HTTP|location|content-type)"
done
Then check three places:
- Content type. You need
text/plain, nottext/html. An HTML placeholder with status 200 hides the problem. - The robots.txt report in Search Console. It shows the last version fetched, the crawl time and errors.
- Crawl stats. The "Host status" block shows whether there were failures fetching robots.txt. A series of failures within a day matches a crawl stop.
And one last step: set up separate monitoring with an alert on the /robots.txt response code. A check every 5 minutes costs 288 requests a day for 77 bytes, while a Search Console report learns about a failure with a delay of days.
Who this is not for
For a site without robots.txt and without a single Disallow (a small brochure site with nothing to block), a 404 at this address does no harm and there is nothing to fix. It is enough that the file does not return 5xx.
The limits of my conclusions: the rules in the table describe Google. Yandex and Bing interpret response codes in their own way, and I did not check their documentation here. The scheme with 12 hours and 30 days comes from the documentation, not from my experiment. For a site with a large catalog where crawl budget is limited, the cost of a crawl stop is higher than for a hundred-page blog. And if you are looking for why pages disappeared from search at all, start with the reasons Google is not indexing your pages: robots.txt is only one of the candidates there. If you want this analysed on your project, I include it in a technical SEO audit.
