Discuss project

Keep Google From Indexing a Page: noindex vs robots.txt

To keep a page out of Google, put noindex on an open page. robots.txt alone does not hide it. I checked 100 big US sites on 16 November 2026.

Vladislav Krivorutško — founder of ADLAB
Vladislav Krivorutško

November 16, 2026 · 14 min read

From the ADLAB channel on YouTube. Watch on YouTube

Contents
  1. 01Short answer
  2. 02Three tools that sound alike
  3. 03What 100 big US sites do
  4. 04The conflict: robots.txt plus noindex
  5. 05What to choose
  6. 06Check one page in a minute
  7. 07Mistakes I keep seeing
  8. 08When robots.txt is still the right tool
  9. Check yourself
  10. Files from the video

TL;DR - key points

  1. 1To keep a normal page out of Google, put noindex on it (a meta tag, or an X-Robots-Tag header for a PDF or an image) and leave the page open to crawling
  2. 2robots.txt stops Google from opening a page, not from listing the URL: Google says a blocked page can still appear in results, for example if other pages link to it
  3. 3A page that is blocked in robots.txt and also carries noindex does not work: Google never opens the page, so it never reads the tag
  4. 4On 16 November 2026 I checked 100 big US sites: 42 showed a service page to a plain request, 23 of those 42 block it in robots.txt, and 8 of the 23 add noindex as well
  5. 5Private page: a login. Deleted page: 404 or 410. Leaked today: the Removals tool in Search Console, then fix the cause, because the block lasts about six months

Short answer

To keep a page out of Google, put noindex on it and leave the page open to crawling. Use a meta tag for a web page and an X-Robots-Tag header for a PDF or an image. A Disallow line in robots.txt is a different tool: it stops Google from opening the page, and Google writes that a blocked page "can still appear in search results, for example if other pages link to it".

The worst result comes from using both on one page. On 16 November 2026 I checked what 100 big US sites do with their search, cart, login and account pages. 42 of them showed me one of those pages. 23 of the 42 rely on robots.txt, and 8 of those 23 also put noindex on the page, a tag Google never reads because it never opens the page. Below: what each tool does, the numbers, a one-minute check and the mistakes I see most often.

Three tools that sound alike

ToolWhat it tells GoogleCan the page still appear in results?Use it for
robots.txt DisallowDo not open this URLYes, for example if other pages link to itSaving crawling on endless URLs
noindex (meta tag or header)Open it, read it, do not list itNo: Google drops the page entirely once it reads the rulePages visitors may open but Google should not list
Login or passwordNobody gets in without credentialsGoogle cannot read what it cannot openAnything private

Google's introduction to robots.txt says the file is not a mechanism for keeping a web page out of Google and names two ways out: noindex or a password. The difference in one line: robots.txt is a sign on the door, noindex is a note inside the room. A visitor who obeys the sign never reads the note.

What 100 big US sites do

The method, so you can repeat it. The sites are the same 100 big US sites in 8 groups that I used for my schema markup measurement. On 16 November 2026 a script requested each site's robots.txt, then four service addresses: internal search (/search?q=test), /cart, /login and /account. It used a plain GET with a Chrome user agent, no JavaScript, and wrote down three things for every address: the HTTP status, whether robots.txt blocks the final address for Googlebot, and whether a noindex sits in the X-Robots-Tag header or in a robots meta tag.

The first finding is about access, not about indexing: 58 of the 100 sites showed me no service page at all. Of the 400 requests, 139 ended in a 404, 94 in a 403, 16 in a 202, 13 in a 429, and 20 got no answer. Big sites close themselves to scripts. Everything below is about the 42 that did answer with a real page.

How 42 big sites keep service pages out of GoogleOpen + noindex (documented way)11 · 26%Open, nothing set8 · 19%Blocked in robots.txt only15 · 36%Blocked + noindex (conflict)8 · 19%Sites out of 42 that showed a readableservice page; green is the documented way
My measurement of 16 November 2026: 100 big US sites, 4 service addresses each, plain request without JavaScript. The sites are grouped by the heaviest category among their readable pages.
  • 11 sites use the documented way: the page is open to crawling and carries noindex. On the search pages of Healthline, WebMD, Investopedia and NPR I found exactly that.
  • 8 sites do nothing: the page is open and has no noindex.
  • 15 sites rely on robots.txt alone. Reddit blocks search, cart, login and account for all crawlers and I found no noindex on any of the four. Target blocks its cart, CNN, Fox News, Wired, BBC and GitHub block search.
  • 8 sites do both: they block the page in robots.txt and put noindex on it. Sephora, NBC News, Delish, Medium, Pinterest, CBS News, CNBC and The Verge, all on the search page.

Add the last two groups: 23 of 42 (55%) keep their service pages behind a robots.txt rule. Look at the search pages only and it is sharper: 21 of the 32 readable search pages are blocked in robots.txt, and all 11 open ones carry noindex.

A correction to my own count. My first count on the same day, with Python's standard robots.txt parser, said 14 of 42. That parser ignores * and $ in rules, loses a group when a blank line follows its User-agent line and does not merge several groups of one agent. I recounted with Google's matching rules (the longest rule wins, groups merge, wildcards work) against robots.txt files read again the same day. The recount moved 9 sites and gave the numbers above. Both verdicts are in the dataset.

Two cautions. Blocking a search page in robots.txt can be a deliberate choice, because search pages are endless and the rule saves crawling. I am not saying these teams are careless; I am saying the rule does not promise to keep the page out of results. And I did not check whether any named page is in Google's index: this is a measurement of settings.

The conflict: robots.txt plus noindex

Take Medium. On 16 November 2026 its robots.txt blocked search with a query for every crawler, and the search page itself carried a noindex tag:

$ curl -s https://medium.com/robots.txt | grep -i search
Disallow: /search?q$
Disallow: /search?q=
Disallow: /*/search?q=
Disallow: /*/search/*?q=

$ curl -s "https://medium.com/search?q=test" | grep -o '<meta name="robots"[^>]*>'
<meta name="robots" content="noindex"/>

On 16 November 2026 my script saw that tag because it does not obey robots.txt. Google does obey it: before it opens a page, it checks the file, finds the Disallow rule above and stops.

Why noindex is not read on a blocked pageGooglebot wants /search?q=testStep 1: reads robots.txtDisallow: /search?q=Stop: the page is not opened<meta name="robots" content="noindex">The tag is there. Nobody reads it.
Order of events for a page that is blocked in robots.txt and carries noindex. Google's documentation: a page blocked by robots.txt means the crawler will never see the noindex rule.

Google's page indexing help states it directly: a robots.txt rule "will actually prevent noindex from being seen by Google". So the second lock is on a door nobody opens, and the result is the same as having no noindex. If other pages link to the URL, it can appear in results, and Search Console lists it as Indexed, though blocked by robots.txt, where Google's advice is to remove the block and use noindex.

For a search page this costs little: all 8 conflicts I found sit on search pages, which have no value in results anyway. The same pattern on a page that matters, such as an old campaign page, a draft or a thank-you page, leaves it as visible as before, and the person who added the tag believes the page is hidden.

What to choose

Your goalUseDo not use
A page visitors may open, Google should not listnoindex meta tag, page open to crawlingA Disallow line
A PDF, image or videoX-Robots-Tag: noindex header on the fileA meta tag (no HTML to hold it)
A private page: account, draft, stagingA login or passwordnoindex or robots.txt alone
A page deleted for goodA 404 or 410A 200 page that says "not found", or a redirect to the homepage
Leaked and must go todayRemovals tool in Search Console, then one of the rows aboveThe tool alone
Save crawling on endless URLsA Disallow ruleCounting on it to hide the pages
Already indexed, should leaveKeep it open, add noindex, wait for a recrawl, block later if you wantBlocking it first

The last row is my working order, not a Google recipe: a blocked page cannot be recrawled, so the tag is never seen. The rest follows Google's documentation, and the cheat sheet below the article has the same table with the quotes and the Apache and nginx snippets for PDFs.

On the removal tool Google is blunt: "A successful request lasts only about six months". For a permanent result the page needs a 404 or 410, a password or a noindex tag. For deleted pages, my article on 404, 410 or redirect covers which one to return and why a redirect to the homepage counts as a soft 404.

Check one page in a minute

  1. Header. curl -s -D - -o /dev/null https://your-site/page and look for x-robots-tag. This sends a GET, which matters: some servers answer a HEAD request (curl -I) differently from a GET.
  2. Meta tag. View the page source and search for noindex, or run curl -s https://your-site/page | grep -io '<meta name="robots"[^>]*>'. Only the tags named robots and googlebot count.
  3. robots.txt. Open https://your-site/robots.txt and look for a rule that matches the page. * and $ work, and the longest matching rule wins.
  4. Both found? That is the conflict. Remove the robots.txt rule and keep the tag.
  5. Google's own view. In Search Console, URL Inspection shows "Indexing allowed?" and names noindex when that is the reason.

For more than a handful of pages I use a script: the indexing check takes a list of URLs and returns one verdict per URL, with conflict for the combination above. Its robots.txt matching follows Google's rules, not Python's standard parser.

Mistakes I keep seeing

  • noindex as protection. A noindex page is hidden from results, not from people: anyone with the link opens it, and Google itself calls noindex the less secure option next to a password or a 404. A staging site belongs behind a login; I describe how in staging without Disallow: /.
  • canonical instead of noindex. A canonical tag is a hint about which duplicate to keep, not a request to drop a page. How Google treats it is in the canonical article.
  • A live page that says "not found" instead of a 404. A deleted page that still answers 200 is a soft 404: the text says it is gone, the server says it is fine. Return 404 or 410.
  • The WordPress setting left on. The checkbox that discourages search engines is meant for a site in development; ticked after launch, it asks search engines to skip the whole site. After a launch, open the page source and /robots.txt of the live site and look for noindex and Disallow: /.
  • noindex pages in the sitemap. A sitemap says "index these"; a noindex page says the opposite. Take them out; the sitemap and robots.txt article has the rules.

When robots.txt is still the right tool

If the goal is to save server load on endless URLs, such as filter combinations or internal search results, robots.txt does exactly that. Then accept that the URL may still appear in results, and decide whether that matters. My article on internal site search walks through that decision for search pages, and crawl budget says when the savings are worth it at all: on a site of a few thousand pages they usually are not.

Who this is not for: if your site has none of the 4 service pages I probed (search, cart, login, account), or they never reach search, there is nothing to fix here. And if a page already has a login, noindex on it is harmless but redundant.

The limits of my numbers:

  • Only the 42 that answered. Big sites close themselves to scripts, so the 42 are sites that answer an unknown client with HTML. The shares are not the share of big sites.
  • Raw HTML. A noindex that a script adds after load was invisible to me, so the share of noindex is a floor.
  • Chrome, not Googlebot. A site can serve Googlebot another robots.txt or another page. I did not check any page against Google's index or Search Console.
  • One day. 16 November 2026, with robots.txt read again the same day. Sites change these files.

If you want the robots.txt, noindex and crawl settings of your own site checked together, I do it as part of a technical SEO audit.

Quiz

Check yourself

A few questions on the article. The answer and the reasoning appear as soon as you pick; nothing is sent anywhere.

0/4

answered

  1. 1.What does a Disallow line in robots.txt do?
  2. 2.Why does noindex fail on a page that is blocked in robots.txt?
  3. 3.How do you keep a PDF out of Google?
  4. 4.A page leaked and has to disappear from results today. What do you do first?

ADLAB · files from the video

Files from the video

Scripts, prompts, templates and data shown in the video. Text files open as plain text; the zip holds every file of one item.

  1. 01Dataset6 files · 64 KB · CC-BY-4.0

    How 100 big US sites keep service pages out of Google (data, 2026-11-16)

    Of 100 large US sites, 42 showed a service page: 23 block it in robots.txt, 8 of them with noindex too. CSV of 400 probes, CC BY 4.0, with the scripts.

    What's inside · 6 files
    • big-sites-indexing-2026.csv

      One row per probe: site, probe, status, final path, robots.txt verdict and rule, noindex in header and meta, site class

      34 KB
    • analyze.py

      Python 3.8+, standard library only; reproduces every number on this page, with an offline self-test

      7.1 KB
    • measure.py

      Repeats the measurement on your own list of sites (requests and beautifulsoup4)

      9.5 KB
    • sites.txt

      The 100 sites with the author's category

      1.8 KB
    • requirements.txt

      The two packages of measure.py

      36 B
    • README.md

      Method, correction of the first count, columns, results, limits

      11 KB
  2. 02Script6 files · 29 KB · MIT

    Indexing check script: robots.txt and noindex for a list of URLs

    Python script: for a list of URLs it shows the robots.txt verdict for Googlebot, noindex in header and meta tag, and flags pages blocked AND noindex.

    What's inside · 6 files
    • README.md

      3-step run, the seven verdicts, CSV columns, how robots.txt is matched, gotchas, Tested on

      7.4 KB
    • indexing_check.py

      The check: Python 3.8+, one package (requests), with an offline self-test of 33 checks

      17 KB
    • requirements.txt

      requests>=2.31, installed into a virtual environment

      15 B
    • urls.sample.txt

      7 public URLs of large sites with the verdicts seen on 2026-11-16

      785 B
    • urls.demo.txt

      9 local URLs covering every verdict, for the demo server

      783 B
    • demo_server.py

      A tiny local site with robots.txt, meta noindex, a PDF with a header and a 410: no internet needed

      2.4 KB
  3. 03Checklist · Template2 files · 10 KB · CC-BY-4.0

    Noindex or robots.txt cheat sheet: which tool for which goal

    One-page sheet: eight goals mapped to noindex, X-Robots-Tag, login, 404/410 or robots.txt, the combinations that cancel each other, snippets and sources.

    What's inside · 2 files
    • noindex-or-robots-cheat-sheet.md

      Goal to tool, what not to combine, meta and header snippets, one-minute check, what 100 large sites do, Google sources

      7.8 KB
    • README.md

      3 steps and what to check before you rely on it

      2.3 KB

Frequently asked questions

How do I stop Google from indexing a page?
Add noindex to the page, as a meta tag in the HTML or as an X-Robots-Tag header, and leave the page open to crawling so that Google can read it. For a PDF or an image use the header, because a file has no HTML to put a tag in. If the page must stay private, put it behind a login instead: noindex hides a page from the results, but anyone with the link can still open it.
Is robots.txt enough to keep a page out of Google?
No. Google's introduction to robots.txt says it is not a mechanism for keeping a web page out of Google. A page blocked there can still appear in results, for example if other pages link to it. Use noindex on an open page, or a login.
Can I use noindex and robots.txt on the same page?
You can, but it does not work. Google checks robots.txt before it opens a page, so on a blocked page it never sees the noindex. In my measurement of 100 big US sites, 8 of the 42 that showed a service page had exactly this combination on their search page. Remove the robots.txt rule and keep the tag.
How long does noindex take to work?
There is no fixed time. The page leaves the results after Google crawls it again and reads the tag, so it depends on how often Google visits that page. URL Inspection in Search Console shows when it was last crawled. If something has to disappear today, use the Removals tool and fix the cause as well.
How do I noindex a PDF or an image?
Send an X-Robots-Tag: noindex header with the file. Google's documentation gives patterns for Apache and nginx that match every .pdf or image file of a site, and my cheat sheet has them ready to copy. Check the result with curl -I, because a header added in the wrong block is silently dropped.
What does Indexed, though blocked by robots.txt mean in Search Console?
Google found the URL, for example through links, and listed it, but could not open the page because robots.txt blocks it. Google's own advice for this status is to remove the robots.txt block and use noindex. The same status is the sign of the mistake described in this article.
Vladislav Krivorutško — founder of ADLAB

About the author

Vladislav Krivorutško

Founder of ADLAB OÜ · SEO and Google Ads

Over 20 years in search traffic and monetization, and on the Estonian market since 2017. I work solo: I run the audit, build the strategy and deliver the project myself — no subcontractors, no templates. I only write about what I have tested on my own and client sites.

  • 20+ years in search traffic
  • 50+ end-to-end projects
  • Own sites in competitive niches
  • SEO for ru/et/en in one market
More about me

Conclusion

A page leaves Google when Google can open it, read noindex and drop it. Every shortcut around that, a Disallow line, a canonical tag, an empty page behind a redirect, changes what Google crawls but not what it lists. Of the 42 big sites that let me read a service page, only 11 did it the documented way, and 8 locked the page in robots.txt and put a tag on it that Google will never read. Check one page today with the three commands above. If the numbers on your own site look like mine, I go through robots.txt, noindex and crawl settings as part of a technical SEO audit.

Read next