The short answer
Robots.txt controls crawling: it tells a crawler which addresses not to fetch. Sitemap.xml controls discovery: it lists the pages you want a search engine to know about. Neither file controls indexing directly — that is handled by the noindex meta tag, canonical and the quality of the page itself.
From this follows the main practical rule: a page you want out of search results must not be blocked in robots.txt. Otherwise the crawler never reads the noindex and the page stays in the index — just without a description.
What robots.txt actually does
Robots.txt sits at the domain root (https://site.ee/robots.txt) and is read before pages are crawled. Inside is a set of rules along the lines of "this crawler does not go here".
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*?sort=
Disallow: /cart
Sitemap: https://site.ee/sitemap.xml
What matters here:
- It is a request, not protection. Major search engines respect it, plenty of scrapers do not. Blocking private data with robots.txt is pointless: the file is public, so you are effectively publishing a list of the addresses you consider sensitive.
- A blocked page can stay in search results. If external links point to it, Google knows the URL and may display it without a snippet. In Search Console this is "Indexed, though blocked by robots.txt".
- Disallow does not remove what is already indexed. Quite the opposite: by blocking the crawl you take away Google's ability to see that the page changed or now carries
noindex. - Rule order is not hierarchical. The most specific rule wins, not the first one from the top.
Allowcan overrideDisallowfor a nested path. - Blocking CSS and JS is an obsolete practice. Google renders pages; block styles and scripts and you show it a broken layout.
One separate line is Sitemap: with the full absolute address of your sitemap. It is the only way to announce a sitemap to search engines where you have no webmaster console.
What belongs in sitemap.xml — and what does not
The most common misconception: a sitemap is "every URL on the site". It is not. It is a list of pages you consider worth indexing. Every URL in it is a statement to a search engine, and statements have to be consistent.
| URL type | In the sitemap? | Why |
|---|---|---|
| Page returns 200 and is indexable | Yes | This is what a sitemap is for |
| Redirecting page (301/302) | No | List the final address instead |
| 404 or 410 | No | A direct signal the file is unmaintained |
Page with noindex | No | Direct conflict of signals |
| Non-canonical version | No | Only the canonical address belongs here |
| URLs with sorting and filter parameters | No | They generate duplicates |
| Category pagination | Usually no | Discovered through links |
| Tag and archive pages | Depends | Only if you genuinely need them in search |
| Language versions | Yes | Each with its own URL, tied together by hreflang |
Format limits: up to 50,000 URLs and up to 50 MB uncompressed per file. Beyond that the sitemap is split and tied together by a sitemap index file. In practice only catalogues reach those limits, but splitting by page type — products, categories, articles — pays off earlier: Search Console then shows you exactly which type indexes badly.
On the optional tags: Google uses lastmod when it can be trusted. priority and changefreq are ignored, according to Google's own staff — do not spend time on them.
How these two files break indexing
Over the years I keep seeing the same short list.
1. A site-wide Disallow left over from development. The line Disallow: / travels from staging to production. Symptom: traffic collapses within a couple of weeks and "Blocked by robots.txt" grows in Search Console. It takes ten seconds to check — and it is the first thing I open when a site "just stopped showing up".
2. Disallow and noindex at the same time. Someone wanted a page out of search results, so they blocked it both in robots.txt and with a meta tag. The result is the opposite of the intent: noindex is never read, the page stays. The correct order is covered in the article on why Google is not indexing your pages.
3. A sitemap nobody has updated. Half the addresses return 404 and none of the new pages are there. Google gradually stops re-reading it and you lose the simplest channel for announcing new material.
4. Using the sitemap to force weak pages into the index. Adding a URL to a sitemap does not make the page worth showing. If Google sees it and passes, the problem is value or duplication, not discovery.
5. A relative address in the Sitemap directive. Only an absolute URL with protocol and domain works.
6. Different robots.txt on www and non-www, http and https. Formally these are different hosts and each reads its own file. This gets forgotten right after redirects are configured.
7. The sitemap itself blocked in robots.txt. Rarer, but it happens — usually when the sitemap lives in a blocked system folder.
What I check in practice
The sequence I run on any site before touching content:
- Open
/robots.txtin a browser. It should return 200 and be short. A hundred-line file almost always contains years of accumulated junk. - Scan for
Disallow: /and anything blocking whole sections. Every such rule needs an explanation. - Check a few important URLs with URL Inspection in Search Console: is crawling allowed, and which address does Google treat as canonical.
- Open the Sitemaps report: sitemap read, date recent, discovered URL count comparable to the real number of pages. A difference of several times over is worth digging into.
- Review the Pages report, specifically "Blocked by robots.txt" and "Discovered – currently not indexed". The first catches excess blocking; the second usually points at quality and structure rather than at these files.
- Compare the sitemap against reality: crawl the site and diff the sitemap URL list against the pages reachable by links. Differences in both directions are informative — what is missing from the sitemap is a candidate for orphan pages, and what is missing from the site is a dead address in the sitemap.
On the limits of all this: on a 20–40 page site with decent internal linking the sitemap decides almost nothing — Google finds everything through links within days. It becomes meaningful in two cases: a fresh domain with no external links, and a large catalogue where some pages sit deep. Expecting ranking gains from it is misplaced entirely: it is a discovery tool, not a ranking one.
If nothing changes within a day of editing robots.txt, that is normal. Google caches the file and does not re-read it on every crawl.
A working minimum for a normal site
The setup I start from on a service site or a small shop:
- Block in robots.txt only CMS system sections, the cart, checkout, the customer account and internal search results.
- Do not block CSS, JS, images or any page that is supposed to rank.
- Declare the absolute sitemap address with the
Sitemap:directive. - Generate sitemap.xml automatically through the CMS or a plugin, never by hand.
- Exclude from the sitemap non-canonical URLs, redirects,
noindexpages and parameterised addresses. - Submit the sitemap in Search Console and come back to the report a week later.
- For junk URLs already in the index, use
noindexorcanonicalwhile crawling stays open — and only block them in robots.txt after a recrawl, if crawl budget justifies it.
If step 7 reveals that parameter and filter URLs multiply by the hundreds, two text files will not fix it: you need URL generation rules at the CMS level, which is web development. And once those junk addresses have pulled impressions onto themselves and compete with real pages, you are dealing with keyword cannibalization, which has to be untangled separately — usually as part of SEO work that rebuilds the structure.
