How to find the sitemap of a website: short answer
Open your-site/robots.txt and look for lines that start with Sitemap:. The site names its own sitemap files there. It works for your site and for a competitor's (69 of 100 big sites had such a line on 17 November 2026), and guessing addresses such as /sitemap.xml only makes sense when there is no such line.
The order matters. On 17 November 2026 I checked 100 big US sites: /sitemap.xml returned a real sitemap on 27 of them, while robots.txt named one on 69. Below: which file to look for, where the sitemap is when robots.txt is silent, how to find a sitemap in a minute and read it, and the mistakes I make most often.
What a sitemap is and which one to look for
By a sitemap I mean the XML file with a list of addresses that a site prepares for search engines, often with a last-modified date, lastmod, next to each. Google's overview of sitemaps describes it as a file with information about the pages, videos and other files of a site. If you mean the HTML "site map" page in a footer, it is made for people and does not show what the site gives to a search engine.
| What you found | What is inside | What to do |
|---|---|---|
Sitemap file (urlset) | A list of <loc> tags with page addresses, sometimes with <lastmod> | Count the addresses, read the dates |
Sitemap index (sitemapindex) | A list of <loc> tags with addresses of other sitemaps | Open the child sitemaps |
| HTML "Site map" page | An ordinary page with links | Almost useless for research |
On a big site you usually meet an index first. Of the 58 sites where I found a readable sitemap, 46 gave an index and only 12 a single file. The biggest index belongs to Medium: 28,315 child sitemap addresses in one file. That is by design: according to Google's guide, one sitemap holds up to 50 MB uncompressed or 50,000 URLs, and a bigger site splits it into parts and joins them with an index.
If you need to create a sitemap rather than find one, read sitemap.xml and robots.txt: what belongs in each: it covers how to create a sitemap and connect it. This article is only about finding and reading.
How to find a site's sitemap: robots.txt against guessing
First the method, so you can repeat it. The sites are the same 100 big US sites as in my schema markup measurement, in 8 groups (retail 20, news 20, recipes 10, reference 10, travel 10, local services 10, tech 10, media 10). On 17 November 2026 a Python script requested robots.txt of every site, wrote down the Sitemap: lines and tested six common addresses. The requests were plain: Chrome user agent, en-US, redirects followed, no JavaScript. It opened the first sitemap it found and recorded the type, compression, the number of <loc> tags and whether lastmod, priority and changefreq are there.
The first finding is about access: robots.txt was readable on 82 of 100 sites. On the other 18 a bot wall answered, or a 404, or something that was not a robots file. 69 of the 82 contain a Sitemap: line, which is 84%.
| Where I looked | Sites out of 100 | Note |
|---|---|---|
Sitemap: line in robots.txt | 69 | Out of 82 sites with a readable robots.txt |
/sitemap.xml | 27 | A real sitemap, not a placeholder |
/sitemap_index.xml | 6 | Counted separately from the other addresses |
/sitemaps/sitemap.xml | 5 | Counted separately |
/wp-sitemap.xml | 3 | The built-in WordPress sitemap |
/sitemap.xml.gz and /sitemap-index.xml | 2 each | Counted separately |
| Any of the six common addresses | 32 | One site can answer on several |
Guessing loses even where nobody hides the sitemap. Of the 69 sites with a robots.txt line, 44 gave no readable sitemap on /sitemap.xml, and on 39 none of the six addresses worked. The reverse is rare: only two sites, HubSpot and Shopify, had no line and were found by guessing.
For Google this is the normal route. A Sitemap: line may stand anywhere in robots.txt, there may be several, and Google finds the sitemap on its next read of the file. I also ran the same script on 30 sites of the credit and casino niches: 22 of 24 readable robots.txt files had a Sitemap: line and /sitemap.xml worked on 17. What Google does when robots.txt answers with a redirect or an error is covered in robots.txt with a redirect, 404 or 5xx.
Sitemap not at /sitemap.xml: what to try next
13 of the 82 sites with a readable robots.txt had no Sitemap: line. That does not mean there is no sitemap. When /sitemap.xml answers "not found", I check in this order:
- Common addresses.
/sitemap.xml,/sitemap_index.xml,/sitemap-index.xml,/wp-sitemap.xml(the sitemap WordPress builds in),/sitemaps/sitemap.xml,/sitemap.xml.gz. A 200 proves nothing: look at what is inside,<urlset>or<sitemapindex>. - Google search. The query
site:domain sitemapsometimes shows an indexed sitemap file. It is a rough tool, and an empty result does not mean there is none. - If the site is yours. Open the Sitemaps report in Search Console: it shows which sitemaps were submitted and when Google read them. In a CMS the sitemap is usually made by a plugin or a built-in function, and its address is in the settings.
A site may have no sitemap at all, and that is allowed. The same Google overview says a small site with good internal links may not need one. If you wonder how to get a sitemap for such a site, the menu and the internal links already do the job.
How to see a competitor's sitemap and read it
The route was the same on each of the 58 big sites where I read a sitemap: from robots.txt to the index, from the index to the child sitemaps, from them to addresses and dates.
Open every Sitemap: line. 45 of the 69 sites list more than one, and Booking.com lists 443. The first line often leads not to the main sitemap: on NYTimes, CNN, The Guardian and The Verge it is a news sitemap, on Nike a help sitemap, on Angi an article sitemap.
From the index, go to the biggest child. Its name usually tells what is inside: products, categories, articles. The number of addresses in it gives a rough size of that section.
Sort the addresses by lastmod. At the top is what the site changed in the last days: new products, rewritten articles, fresh categories. The date is not everywhere: lastmod appeared in 32 of the 58 sitemaps. According to Google's guide it is used only when the value is consistently and verifiably accurate, and a changed year in the footer does not count as an update.
Skip priority and changefreq. Google ignores both. In my check priority appeared in 2 of 58 sitemaps (Nike and HubSpot) and changefreq in 3 (Nike, HubSpot and The Guardian); to me they are a trace of an old plugin. One caveat: the script looked for the tag in the first file it opened, and that was the file from the first robots.txt line, so on the main sitemap of NYTimes or Nike the picture may differ.
My opinion, not a measured fact: a sitemap, even one with 28,315 child sitemaps like Medium's, shows which pages a site wants indexed, not which of them rank. Google itself writes that a sitemap does not guarantee that all its items will be crawled and indexed. If a competitor's sitemap lists pages with noindex, the site contradicts itself; why that fails is in my article on noindex or robots.txt.
How to get a website's sitemap in one minute
Five terminal commands; put the domain in place of example.com.
- Lines in robots.txt:
curl -s https://example.com/robots.txt | grep -i '^sitemap:' - What is at an address:
curl -sL https://example.com/sitemap.xml | head -c 300. You want<urlsetor<sitemapindex;<htmlmeans a placeholder. - A compressed sitemap:
curl -sL https://example.com/sitemap.xml.gz | gunzip | head -c 300 - How many addresses:
curl -sL SITEMAP-URL | grep -o '<loc>' | wc -l. Count the tags, not the lines: many sitemaps are written on one line. - The freshest dates:
curl -sL SITEMAP-URL | grep -o '<lastmod>[^<]*' | sort -r | head
When there are more than a couple of domains, I run the sitemap-finder script: it takes a list of domains and writes to a CSV the sitemap it found, how (robots.txt or a common address), index or not, the number of addresses and whether lastmod, priority and changefreq are present.
Mistakes I make most often
All examples come from the same check of 17 November 2026. Each of these mistakes can make you think a site has no sitemap when it does.
- Only the first
Sitemap:line. 45 of the 69 sites list several, and the first often leads to a news or help sitemap. Open all of them. - HTML instead of XML. 11 sites, among them Walmart, Wikipedia and Pinterest, answered a plain request with HTML instead of a sitemap. That is bot protection, not a missing sitemap: open the address in a browser.
- A compressed
.gzfile. Target, NYTimes and Airbnb compress theirs with gzip. A browser will not show it but download it, and you have to unpack it. - A sitemap on another host. NPR keeps it on googlecrawl.npr.org, Cleveland Clinic on my.clevelandclinic.org, Zillow on mortgageapi.zillow.com. A script that drops other hosts will lose such a sitemap.
- An index instead of addresses. An index holds sitemap addresses, not pages: Medium's 28,315 is the number of child sitemaps. Count pages in the child files.
Who this is not for, and the limits of my numbers
If the site is yours and connected to Search Console, you need not look for the sitemap: the Sitemaps report shows the address and the date of the last read. Research through sitemaps is for someone else's site, or for a site you inherited without documentation.
The limits of my numbers:
- Big US sites close themselves to scripts. That is why robots.txt was read on 82 of 100 and a sitemap on 58. Of the rest, 29 had none found, 11 answered with HTML and 2 with something else.
- Raw response, no JavaScript, Chrome and not Googlebot. A site can show Googlebot something else.
- The tag exists, but is its date right? I did not check whether the date in
lastmodis true. - A sitemap shows intent. A page can be in it and out of the index, and the other way round; I compared nothing with Google's index.
- One day. 17 November 2026. Sites change robots.txt and sitemaps.
If you want the sitemap, robots.txt and crawling of your own site checked together, I do that as part of a technical SEO audit.
Files for this article
Below the article: the sitemap-finder script, a cheat sheet with the search order, the addresses, the site: operator and the Search Console report, and the dataset: a CSV of 100 big sites under CC BY 4.0 with the measuring script.
