Short answer
To keep a page out of Google, put noindex on it and leave the page open to crawling. Use a meta tag for a web page and an X-Robots-Tag header for a PDF or an image. A Disallow line in robots.txt is a different tool: it stops Google from opening the page, and Google writes that a blocked page "can still appear in search results, for example if other pages link to it".
The worst result comes from using both on one page. On 16 November 2026 I checked what 100 big US sites do with their search, cart, login and account pages. 42 of them showed me one of those pages. 23 of the 42 rely on robots.txt, and 8 of those 23 also put noindex on the page, a tag Google never reads because it never opens the page. Below: what each tool does, the numbers, a one-minute check and the mistakes I see most often.
Three tools that sound alike
| Tool | What it tells Google | Can the page still appear in results? | Use it for |
|---|---|---|---|
robots.txt Disallow | Do not open this URL | Yes, for example if other pages link to it | Saving crawling on endless URLs |
noindex (meta tag or header) | Open it, read it, do not list it | No: Google drops the page entirely once it reads the rule | Pages visitors may open but Google should not list |
| Login or password | Nobody gets in without credentials | Google cannot read what it cannot open | Anything private |
Google's introduction to robots.txt says the file is not a mechanism for keeping a web page out of Google and names two ways out: noindex or a password. The difference in one line: robots.txt is a sign on the door, noindex is a note inside the room. A visitor who obeys the sign never reads the note.
What 100 big US sites do
The method, so you can repeat it. The sites are the same 100 big US sites in 8 groups that I used for my schema markup measurement. On 16 November 2026 a script requested each site's robots.txt, then four service addresses: internal search (/search?q=test), /cart, /login and /account. It used a plain GET with a Chrome user agent, no JavaScript, and wrote down three things for every address: the HTTP status, whether robots.txt blocks the final address for Googlebot, and whether a noindex sits in the X-Robots-Tag header or in a robots meta tag.
The first finding is about access, not about indexing: 58 of the 100 sites showed me no service page at all. Of the 400 requests, 139 ended in a 404, 94 in a 403, 16 in a 202, 13 in a 429, and 20 got no answer. Big sites close themselves to scripts. Everything below is about the 42 that did answer with a real page.
- 11 sites use the documented way: the page is open to crawling and carries
noindex. On the search pages of Healthline, WebMD, Investopedia and NPR I found exactly that. - 8 sites do nothing: the page is open and has no
noindex. - 15 sites rely on robots.txt alone. Reddit blocks search, cart, login and account for all crawlers and I found no
noindexon any of the four. Target blocks its cart, CNN, Fox News, Wired, BBC and GitHub block search. - 8 sites do both: they block the page in robots.txt and put
noindexon it. Sephora, NBC News, Delish, Medium, Pinterest, CBS News, CNBC and The Verge, all on the search page.
Add the last two groups: 23 of 42 (55%) keep their service pages behind a robots.txt rule. Look at the search pages only and it is sharper: 21 of the 32 readable search pages are blocked in robots.txt, and all 11 open ones carry noindex.
A correction to my own count. My first count on the same day, with Python's standard robots.txt parser, said 14 of 42. That parser ignores * and $ in rules, loses a group when a blank line follows its User-agent line and does not merge several groups of one agent. I recounted with Google's matching rules (the longest rule wins, groups merge, wildcards work) against robots.txt files read again the same day. The recount moved 9 sites and gave the numbers above. Both verdicts are in the dataset.
Two cautions. Blocking a search page in robots.txt can be a deliberate choice, because search pages are endless and the rule saves crawling. I am not saying these teams are careless; I am saying the rule does not promise to keep the page out of results. And I did not check whether any named page is in Google's index: this is a measurement of settings.
The conflict: robots.txt plus noindex
Take Medium. On 16 November 2026 its robots.txt blocked search with a query for every crawler, and the search page itself carried a noindex tag:
$ curl -s https://medium.com/robots.txt | grep -i search
Disallow: /search?q$
Disallow: /search?q=
Disallow: /*/search?q=
Disallow: /*/search/*?q=
$ curl -s "https://medium.com/search?q=test" | grep -o '<meta name="robots"[^>]*>'
<meta name="robots" content="noindex"/>
On 16 November 2026 my script saw that tag because it does not obey robots.txt. Google does obey it: before it opens a page, it checks the file, finds the Disallow rule above and stops.
Google's page indexing help states it directly: a robots.txt rule "will actually prevent noindex from being seen by Google". So the second lock is on a door nobody opens, and the result is the same as having no noindex. If other pages link to the URL, it can appear in results, and Search Console lists it as Indexed, though blocked by robots.txt, where Google's advice is to remove the block and use noindex.
For a search page this costs little: all 8 conflicts I found sit on search pages, which have no value in results anyway. The same pattern on a page that matters, such as an old campaign page, a draft or a thank-you page, leaves it as visible as before, and the person who added the tag believes the page is hidden.
What to choose
| Your goal | Use | Do not use |
|---|---|---|
| A page visitors may open, Google should not list | noindex meta tag, page open to crawling | A Disallow line |
| A PDF, image or video | X-Robots-Tag: noindex header on the file | A meta tag (no HTML to hold it) |
| A private page: account, draft, staging | A login or password | noindex or robots.txt alone |
| A page deleted for good | A 404 or 410 | A 200 page that says "not found", or a redirect to the homepage |
| Leaked and must go today | Removals tool in Search Console, then one of the rows above | The tool alone |
| Save crawling on endless URLs | A Disallow rule | Counting on it to hide the pages |
| Already indexed, should leave | Keep it open, add noindex, wait for a recrawl, block later if you want | Blocking it first |
The last row is my working order, not a Google recipe: a blocked page cannot be recrawled, so the tag is never seen. The rest follows Google's documentation, and the cheat sheet below the article has the same table with the quotes and the Apache and nginx snippets for PDFs.
On the removal tool Google is blunt: "A successful request lasts only about six months". For a permanent result the page needs a 404 or 410, a password or a noindex tag. For deleted pages, my article on 404, 410 or redirect covers which one to return and why a redirect to the homepage counts as a soft 404.
Check one page in a minute
- Header.
curl -s -D - -o /dev/null https://your-site/pageand look forx-robots-tag. This sends a GET, which matters: some servers answer a HEAD request (curl -I) differently from a GET. - Meta tag. View the page source and search for
noindex, or runcurl -s https://your-site/page | grep -io '<meta name="robots"[^>]*>'. Only the tags namedrobotsandgooglebotcount. - robots.txt. Open
https://your-site/robots.txtand look for a rule that matches the page.*and$work, and the longest matching rule wins. - Both found? That is the conflict. Remove the robots.txt rule and keep the tag.
- Google's own view. In Search Console, URL Inspection shows "Indexing allowed?" and names noindex when that is the reason.
For more than a handful of pages I use a script: the indexing check takes a list of URLs and returns one verdict per URL, with conflict for the combination above. Its robots.txt matching follows Google's rules, not Python's standard parser.
Mistakes I keep seeing
- noindex as protection. A noindex page is hidden from results, not from people: anyone with the link opens it, and Google itself calls noindex the less secure option next to a password or a 404. A staging site belongs behind a login; I describe how in staging without Disallow: /.
- canonical instead of noindex. A canonical tag is a hint about which duplicate to keep, not a request to drop a page. How Google treats it is in the canonical article.
- A live page that says "not found" instead of a 404. A deleted page that still answers 200 is a soft 404: the text says it is gone, the server says it is fine. Return 404 or 410.
- The WordPress setting left on. The checkbox that discourages search engines is meant for a site in development; ticked after launch, it asks search engines to skip the whole site. After a launch, open the page source and
/robots.txtof the live site and look fornoindexandDisallow: /. - noindex pages in the sitemap. A sitemap says "index these"; a noindex page says the opposite. Take them out; the sitemap and robots.txt article has the rules.
When robots.txt is still the right tool
If the goal is to save server load on endless URLs, such as filter combinations or internal search results, robots.txt does exactly that. Then accept that the URL may still appear in results, and decide whether that matters. My article on internal site search walks through that decision for search pages, and crawl budget says when the savings are worth it at all: on a site of a few thousand pages they usually are not.
Who this is not for: if your site has none of the 4 service pages I probed (search, cart, login, account), or they never reach search, there is nothing to fix here. And if a page already has a login, noindex on it is harmless but redundant.
The limits of my numbers:
- Only the 42 that answered. Big sites close themselves to scripts, so the 42 are sites that answer an unknown client with HTML. The shares are not the share of big sites.
- Raw HTML. A
noindexthat a script adds after load was invisible to me, so the share ofnoindexis a floor. - Chrome, not Googlebot. A site can serve Googlebot another robots.txt or another page. I did not check any page against Google's index or Search Console.
- One day. 16 November 2026, with robots.txt read again the same day. Sites change these files.
If you want the robots.txt, noindex and crawl settings of your own site checked together, I do it as part of a technical SEO audit.
