Staging vs Production: Keep Disallow: / Off Your Live Site

Short answer

Staging is blocked with a password in the web server config (basic auth in nginx or Apache), not with a Disallow: / line in robots.txt. The password lives on the test server and never enters the repository, so a deploy cannot carry it to the live site. The robots.txt file and the noindex meta tag sit in project code and ship to production with everything else.

After every release the live domain needs three checks: the contents of robots.txt, a search for noindex in the homepage source, and the server response headers via curl. The whole thing takes 5 minutes.

Why Disallow: / does not protect a staging site

Disallow: / on staging has two flaws. First, it does not block indexing. Google Search Central states plainly that robots.txt controls crawling and that a blocked URL can still be indexed if links point to it. Staging subdomains get discovered through a link in an email, a task tracker with a public board, a screenshot, a Referer header. Then dev.site.com shows up in results with a note that no description is available.

The second flaw costs more. The file sits in the repository and gets copied to production on release. The site keeps working: pages load, forms submit, nobody notices anything in the browser. People find out one or two weeks later from the impressions graph in Search Console. In my website migration SEO checklist I called this the most common reason a migration fails, and years of audits have not changed that.

How to lock staging so the block cannot travel

I use one setup: basic auth at the nginx level, only in the staging environment config. A Next.js or WordPress project then has no indexing-related line that differs between staging and production.

server {
    server_name dev.example.com;
    auth_basic "Staging";
    auth_basic_user_file /etc/nginx/.htpasswd-staging;
    # ...
}

If the site runs on Cloudflare, Cloudflare Access on the staging subdomain does the same job: access by email or one-time code, again outside the repository. On Vercel, preview deployments send X-Robots-Tag: noindex by default, but the header is not added on your production custom domain. The risk there is of a different kind: the preview you sent the client as an "almost finished site" is the one that is blocked.

When a meta tag in code is unavoidable (say, one Next.js template builds every environment), noindex should switch on only with an explicit variable such as SITE_ENV=staging. A missing variable must mean "index", not the other way round. With inverted logic, the first server where somebody forgot to set the variable ends up blocked.

Where the block hides after a release

When traffic drops after a release, I check 5 places in this order: from the most frequent to the least visible, from robots.txt to Cloudflare rules.

1. The robots.txt file. I open https://site.com/robots.txt in a browser and look for Disallow: / with nothing after the slash. If the file is generated (in Next.js that is app/robots.ts), I also read the generator: environment conditions with swapped branches turn up there.
2. The robots meta tag. I search for noindex in the source of the homepage and one inner page, using "View page source" rather than DevTools. Different templates can behave differently.
3. The X-Robots-Tag header. curl -I https://site.com/ prints the response headers. This block is invisible in the page source and in the browser, which is why people look for it last, even though checking takes a second.
4. The WordPress setting. The "Discourage search engines from indexing this site" checkbox is stored in the database as the blog_public option. Copy the database from staging to production and it comes along, even if the project files were uploaded clean. Since version 5.7 WordPress outputs a noindex meta tag in that case, so check 2 catches it too.
5. CDN and hosting rules. Cloudflare Transform Rules, headers set in a hosting panel, security plugins. If curl -I shows X-Robots-Tag and your nginx config does not contain it, look here.

One case people forget: robots.txt is open, but it blocks the path to CSS and JS. The site is indexed, yet Google renders it without styles. What to block in that file and what to leave open is covered in my article on sitemap.xml and robots.txt best practices.

The five-minute post-release check

I run these 5 steps by hand after any release that touched templates, the nginx config or the database.

1. https://site.com/robots.txt in a browser: no Disallow: /, a Sitemap: line with a working URL.
2. Source of the homepage and one product page or article: no noindex, and canonical points to the live domain, not to dev..
3. curl -I https://site.com/: status 200, no X-Robots-Tag: noindex.
4. URL Inspection in Search Console for the homepage with "Test live URL": status "URL is available to Google".
5. Two or three days later, the Page indexing report: neither "Blocked by robots.txt" nor "Excluded by 'noindex' tag" is growing.

The canonical step is not there for completeness. A Next.js or WordPress template built on staging sometimes takes the domain from an environment variable, and production ends up with a canonical pointing at the test address. Google then sees two versions of the same text and picks the canonical itself; how that looks in the reports is described in my guide to duplicate content.

If the block already reached production

What to do depends on which block fired: robots.txt or noindex. So diagnose with the 5 steps above first, then fix it in Search Console.

• Disallow: / shipped. Fix the file, open the robots.txt report in Search Console and request a recrawl. Google caches robots.txt for up to 24 hours, and without the request the crawler may keep following the old version for another day.
• noindex shipped. Remove it and send the homepage and main sections to URL Inspection. The rest follows during normal crawling.
• In both cases resubmit sitemap.xml: it gives the crawler a list of URLs to revisit.

Recovery time cannot be predicted; it depends on how often the specific site gets crawled. On a site Googlebot visits daily, the first pages return sooner than on a 30-page site the crawler visits once a week. If after two or three weeks pages still sit in "Crawled – currently not indexed", the block is no longer the cause, and the next step is my breakdown of why Google is not indexing your pages.

When basic auth is the wrong choice

A staging password gets in the way in 3 situations, and there I pick Cloudflare Access, an IP allowlist or a separate test URL instead.

• A client or content editor reviews staging on a phone and gets stuck at the login prompt. Cloudflare Access with email login works better here.
• An external service must reach staging without a password: a payment gateway sends webhooks, uptime monitoring checks availability. Then use an IP allowlist with an exception for the service's addresses rather than removing protection.
• You need to see how Google renders a page before release. URL Inspection cannot get past a password. That test runs on a separate open URL with noindex that is deleted right after the test, and it is the only place where a meta tag on a test environment makes sense.