Discuss project

How to Find the Sitemap of a Website: robots.txt First

How to find the sitemap of a website: read robots.txt first, guess addresses second. In my check of 100 big US sites /sitemap.xml worked on only 27.

Vladislav Krivorutško — founder of ADLAB
Vladislav Krivorutško

November 17, 2026 · 14 min read

From the ADLAB channel on YouTube. Watch on YouTube

Contents
  1. 01How to find the sitemap of a website: short answer
  2. 02What a sitemap is and which one to look for
  3. 03How to find a site's sitemap: robots.txt against guessing
  4. 04Sitemap not at /sitemap.xml: what to try next
  5. 05How to see a competitor's sitemap and read it
  6. 06How to get a website's sitemap in one minute
  7. 07Mistakes I make most often
  8. 08Who this is not for, and the limits of my numbers
  9. 09Files for this article
  10. Check yourself
  11. Files from the video

TL;DR - key points

  1. 1To find the sitemap of a website, open its robots.txt and read every line that starts with Sitemap: before you guess any address
  2. 2On 17 November 2026 I checked 100 big US sites: robots.txt named a sitemap on 69, while /sitemap.xml returned a real sitemap on only 27
  3. 3Of the 69 sites with a Sitemap line, 44 gave no readable sitemap on /sitemap.xml, and on 39 none of six common addresses worked
  4. 4Open every Sitemap line, not just the first: 45 of the 69 list more than one, and the first often leads to a news or help file
  5. 5Read the addresses and lastmod inside; Google ignores priority and changefreq, and a sitemap does not guarantee that a page gets indexed

How to find the sitemap of a website: short answer

Open your-site/robots.txt and look for lines that start with Sitemap:. The site names its own sitemap files there. It works for your site and for a competitor's (69 of 100 big sites had such a line on 17 November 2026), and guessing addresses such as /sitemap.xml only makes sense when there is no such line.

The order matters. On 17 November 2026 I checked 100 big US sites: /sitemap.xml returned a real sitemap on 27 of them, while robots.txt named one on 69. Below: which file to look for, where the sitemap is when robots.txt is silent, how to find a sitemap in a minute and read it, and the mistakes I make most often.

What a sitemap is and which one to look for

By a sitemap I mean the XML file with a list of addresses that a site prepares for search engines, often with a last-modified date, lastmod, next to each. Google's overview of sitemaps describes it as a file with information about the pages, videos and other files of a site. If you mean the HTML "site map" page in a footer, it is made for people and does not show what the site gives to a search engine.

What you foundWhat is insideWhat to do
Sitemap file (urlset)A list of <loc> tags with page addresses, sometimes with <lastmod>Count the addresses, read the dates
Sitemap index (sitemapindex)A list of <loc> tags with addresses of other sitemapsOpen the child sitemaps
HTML "Site map" pageAn ordinary page with linksAlmost useless for research

On a big site you usually meet an index first. Of the 58 sites where I found a readable sitemap, 46 gave an index and only 12 a single file. The biggest index belongs to Medium: 28,315 child sitemap addresses in one file. That is by design: according to Google's guide, one sitemap holds up to 50 MB uncompressed or 50,000 URLs, and a bigger site splits it into parts and joins them with an index.

If you need to create a sitemap rather than find one, read sitemap.xml and robots.txt: what belongs in each: it covers how to create a sitemap and connect it. This article is only about finding and reading.

How to find a site's sitemap: robots.txt against guessing

First the method, so you can repeat it. The sites are the same 100 big US sites as in my schema markup measurement, in 8 groups (retail 20, news 20, recipes 10, reference 10, travel 10, local services 10, tech 10, media 10). On 17 November 2026 a Python script requested robots.txt of every site, wrote down the Sitemap: lines and tested six common addresses. The requests were plain: Chrome user agent, en-US, redirects followed, no JavaScript. It opened the first sitemap it found and recorded the type, compression, the number of <loc> tags and whether lastmod, priority and changefreq are there.

The first finding is about access: robots.txt was readable on 82 of 100 sites. On the other 18 a bot wall answered, or a 404, or something that was not a robots file. 69 of the 82 contain a Sitemap: line, which is 84%.

Guessing the address against robots.txt/sitemap.xml returns a sitemap27 of 100robots.txt names a sitemap69 of 100On 44 of those 69, /sitemap.xml gaveno readable sitemap
My check of 17 November 2026: 100 big US sites, plain request without JavaScript. On 18 sites robots.txt could not be read, and they are counted as "no".
Where I lookedSites out of 100Note
Sitemap: line in robots.txt69Out of 82 sites with a readable robots.txt
/sitemap.xml27A real sitemap, not a placeholder
/sitemap_index.xml6Counted separately from the other addresses
/sitemaps/sitemap.xml5Counted separately
/wp-sitemap.xml3The built-in WordPress sitemap
/sitemap.xml.gz and /sitemap-index.xml2 eachCounted separately
Any of the six common addresses32One site can answer on several

Guessing loses even where nobody hides the sitemap. Of the 69 sites with a robots.txt line, 44 gave no readable sitemap on /sitemap.xml, and on 39 none of the six addresses worked. The reverse is rare: only two sites, HubSpot and Shopify, had no line and were found by guessing.

For Google this is the normal route. A Sitemap: line may stand anywhere in robots.txt, there may be several, and Google finds the sitemap on its next read of the file. I also ran the same script on 30 sites of the credit and casino niches: 22 of 24 readable robots.txt files had a Sitemap: line and /sitemap.xml worked on 17. What Google does when robots.txt answers with a redirect or an error is covered in robots.txt with a redirect, 404 or 5xx.

Sitemap not at /sitemap.xml: what to try next

13 of the 82 sites with a readable robots.txt had no Sitemap: line. That does not mean there is no sitemap. When /sitemap.xml answers "not found", I check in this order:

  1. Common addresses. /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml (the sitemap WordPress builds in), /sitemaps/sitemap.xml, /sitemap.xml.gz. A 200 proves nothing: look at what is inside, <urlset> or <sitemapindex>.
  2. Google search. The query site:domain sitemap sometimes shows an indexed sitemap file. It is a rough tool, and an empty result does not mean there is none.
  3. If the site is yours. Open the Sitemaps report in Search Console: it shows which sitemaps were submitted and when Google read them. In a CMS the sitemap is usually made by a plugin or a built-in function, and its address is in the settings.

A site may have no sitemap at all, and that is allowed. The same Google overview says a small site with good internal links may not need one. If you wonder how to get a sitemap for such a site, the menu and the internal links already do the job.

How to see a competitor's sitemap and read it

The route was the same on each of the 58 big sites where I read a sitemap: from robots.txt to the index, from the index to the child sitemaps, from them to addresses and dates.

The route from robots.txt to page addressesrobots.txtSitemap: lines · on 69 of 100 sitesSitemap indexsitemapindex · 46 of 58 readableChild sitemapsproducts, articles, categoriesPage addresses + lastmodlastmod present in 32 of 58
My check of 17 November 2026, 100 big US sites. The shares of indexes and lastmod are counted among the 58 sites with a readable sitemap.

Open every Sitemap: line. 45 of the 69 sites list more than one, and Booking.com lists 443. The first line often leads not to the main sitemap: on NYTimes, CNN, The Guardian and The Verge it is a news sitemap, on Nike a help sitemap, on Angi an article sitemap.

From the index, go to the biggest child. Its name usually tells what is inside: products, categories, articles. The number of addresses in it gives a rough size of that section.

Sort the addresses by lastmod. At the top is what the site changed in the last days: new products, rewritten articles, fresh categories. The date is not everywhere: lastmod appeared in 32 of the 58 sitemaps. According to Google's guide it is used only when the value is consistently and verifiably accurate, and a changed year in the footer does not count as an update.

Skip priority and changefreq. Google ignores both. In my check priority appeared in 2 of 58 sitemaps (Nike and HubSpot) and changefreq in 3 (Nike, HubSpot and The Guardian); to me they are a trace of an old plugin. One caveat: the script looked for the tag in the first file it opened, and that was the file from the first robots.txt line, so on the main sitemap of NYTimes or Nike the picture may differ.

My opinion, not a measured fact: a sitemap, even one with 28,315 child sitemaps like Medium's, shows which pages a site wants indexed, not which of them rank. Google itself writes that a sitemap does not guarantee that all its items will be crawled and indexed. If a competitor's sitemap lists pages with noindex, the site contradicts itself; why that fails is in my article on noindex or robots.txt.

How to get a website's sitemap in one minute

Five terminal commands; put the domain in place of example.com.

  1. Lines in robots.txt: curl -s https://example.com/robots.txt | grep -i '^sitemap:'
  2. What is at an address: curl -sL https://example.com/sitemap.xml | head -c 300. You want <urlset or <sitemapindex; <html means a placeholder.
  3. A compressed sitemap: curl -sL https://example.com/sitemap.xml.gz | gunzip | head -c 300
  4. How many addresses: curl -sL SITEMAP-URL | grep -o '<loc>' | wc -l. Count the tags, not the lines: many sitemaps are written on one line.
  5. The freshest dates: curl -sL SITEMAP-URL | grep -o '<lastmod>[^<]*' | sort -r | head

When there are more than a couple of domains, I run the sitemap-finder script: it takes a list of domains and writes to a CSV the sitemap it found, how (robots.txt or a common address), index or not, the number of addresses and whether lastmod, priority and changefreq are present.

Mistakes I make most often

All examples come from the same check of 17 November 2026. Each of these mistakes can make you think a site has no sitemap when it does.

  • Only the first Sitemap: line. 45 of the 69 sites list several, and the first often leads to a news or help sitemap. Open all of them.
  • HTML instead of XML. 11 sites, among them Walmart, Wikipedia and Pinterest, answered a plain request with HTML instead of a sitemap. That is bot protection, not a missing sitemap: open the address in a browser.
  • A compressed .gz file. Target, NYTimes and Airbnb compress theirs with gzip. A browser will not show it but download it, and you have to unpack it.
  • A sitemap on another host. NPR keeps it on googlecrawl.npr.org, Cleveland Clinic on my.clevelandclinic.org, Zillow on mortgageapi.zillow.com. A script that drops other hosts will lose such a sitemap.
  • An index instead of addresses. An index holds sitemap addresses, not pages: Medium's 28,315 is the number of child sitemaps. Count pages in the child files.

Who this is not for, and the limits of my numbers

If the site is yours and connected to Search Console, you need not look for the sitemap: the Sitemaps report shows the address and the date of the last read. Research through sitemaps is for someone else's site, or for a site you inherited without documentation.

The limits of my numbers:

  • Big US sites close themselves to scripts. That is why robots.txt was read on 82 of 100 and a sitemap on 58. Of the rest, 29 had none found, 11 answered with HTML and 2 with something else.
  • Raw response, no JavaScript, Chrome and not Googlebot. A site can show Googlebot something else.
  • The tag exists, but is its date right? I did not check whether the date in lastmod is true.
  • A sitemap shows intent. A page can be in it and out of the index, and the other way round; I compared nothing with Google's index.
  • One day. 17 November 2026. Sites change robots.txt and sitemaps.

If you want the sitemap, robots.txt and crawling of your own site checked together, I do that as part of a technical SEO audit.

Files for this article

Below the article: the sitemap-finder script, a cheat sheet with the search order, the addresses, the site: operator and the Search Console report, and the dataset: a CSV of 100 big sites under CC BY 4.0 with the measuring script.

Quiz

Check yourself

A few questions on the article. The answer and the reasoning appear as soon as you pick; nothing is sent anywhere.

0/4

answered

  1. 1.Where do you look first for the sitemap of someone else's site?
  2. 2.What is a sitemap index?
  3. 3.Which sitemap tags does Google ignore?
  4. 4.robots.txt has no Sitemap: line. What do you do?

ADLAB · files from the video

Files from the video

Scripts, prompts, templates and data shown in the video. Text files open as plain text; the zip holds every file of one item.

  1. 01Dataset6 files · 25 KB · CC-BY-4.0

    Big sites and sitemaps: where 100 US sites keep theirs (data, 2026-11-17)

    Of 100 large US sites, robots.txt names a sitemap on 69 and /sitemap.xml works on 27. CSV of 100 sites, CC BY 4.0, with the measuring script and analysis.

    What's inside · 6 files
    • big-sites-sitemaps-2026.csv

      One row per site: robots.txt, Sitemap lines, first sitemap, index or not, URL count, lastmod, priority, changefreq

      9.5 KB
    • analyze.py

      Python 3.8+, standard library only; reproduces every number on this page, with an offline self-test

      5.0 KB
    • measure.py

      Repeats the measurement on your own list of sites (requests)

      4.1 KB
    • sites.txt

      The 100 sites with the author's category

      1.8 KB
    • requirements.txt

      The one package of measure.py

      15 B
    • README.md

      Results, method, columns, limits, how to repeat it, citation

      5.0 KB
  2. 02Script5 files · 15 KB · MIT

    Sitemap finder script: find the sitemap of any list of sites

    Python script: for a list of domains it reads robots.txt first, then guesses six addresses, and writes a CSV row: where the sitemap is, index or not, URL count.

    What's inside · 5 files
    • README.md

      3-step run, CSV columns, gotchas (first sitemap is not always the main one), Tested on

      4.4 KB
    • sitemap_finder.py

      The finder: Python 3.8+, one package (requests), with an offline self-test of 17 checks

      10 KB
    • requirements.txt

      requests>=2.31, installed into a virtual environment

      15 B
    • domains.sample.txt

      5 public domains of large sites to see the output

      110 B
    • sample-output.csv

      What the sample run gave on 2026-11-17

      612 B
  3. 03Checklist2 files · 6.9 KB · CC-BY-4.0

    Where to look for a sitemap: cheat sheet with addresses and tag values

    One-page sheet: the order to find a sitemap, six addresses with hit counts, platform defaults, what Google does with lastmod, priority and changefreq, limits.

    What's inside · 2 files
    • sitemap-where-to-look-cheat-sheet.md

      Search order, six addresses with hit counts, platform defaults, tag values, limits, traps, one-minute check of your own site

      5.6 KB
    • README.md

      3 steps and what to check before you rely on it

      1.2 KB

Frequently asked questions

How do I find the sitemap of any website?
Add /robots.txt to the address of the site and look for lines that start with Sitemap:. The site names its own sitemap files there. In my check of 100 big US sites, 69 had such a line, and 45 of those 69 had more than one, so open each. If there are no lines, try common addresses such as /sitemap.xml and /sitemap_index.xml.
Where is sitemap.xml located?
Usually at the root of the site, but the address can be anything: Google lets you name the file as you like and point to it with a Sitemap: line in robots.txt. In my check /sitemap.xml returned a real sitemap on 27 of 100 big sites. Some sitemaps sit on another host name: NPR keeps its on googlecrawl.npr.org and Zillow on mortgageapi.zillow.com.
What if the sitemap is not at /sitemap.xml?
Read robots.txt first: on 44 of the 69 sites with a Sitemap line, /sitemap.xml gave no readable sitemap although the site has one. Then try /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemaps/sitemap.xml and /sitemap.xml.gz. For your own site, open the Sitemaps report in Search Console. A site may also have no sitemap: Google says a small, well-linked site may not need one.
How do I see a competitor's sitemap?
The same way as your own: robots.txt, the Sitemap lines, then the index and the child sitemaps. Open the biggest child, count the addresses and sort them by lastmod to see what changed in the last days. It shows which pages the competitor wants indexed, not which of them rank.
Does Google use priority and changefreq in a sitemap?
No. Google's documentation says it ignores both values. It uses lastmod when the value is consistently and verifiably accurate, and a changed date in the footer does not count as an update. In my check priority appeared in only 2 of 58 readable sitemaps, Nike and HubSpot.
Does a sitemap guarantee that Google indexes my pages?
No. Google states that a sitemap does not guarantee that all the items in it will be crawled and indexed. A page can be in the sitemap and out of the index, and the other way round. So a competitor's sitemap shows the intent of the site, not its rankings.
Vladislav Krivorutško — founder of ADLAB

About the author

Vladislav Krivorutško

Founder of ADLAB OÜ · SEO and Google Ads

Over 20 years in search traffic and monetization, and on the Estonian market since 2017. I work solo: I run the audit, build the strategy and deliver the project myself — no subcontractors, no templates. I only write about what I have tested on my own and client sites.

  • 20+ years in search traffic
  • 50+ end-to-end projects
  • Own sites in competitive niches
  • SEO for ru/et/en in one market
More about me

Conclusion

The sitemap is usually named in robots.txt, so start there instead of guessing. In my check robots.txt named a sitemap on 69 of 100 big sites and /sitemap.xml worked on 27; only two sites, HubSpot and Shopify, were found by guessing alone. Then open every Sitemap line, the index, the biggest child sitemap, and sort the addresses by lastmod. Remember that a sitemap shows which pages a site wants indexed, not which of them rank. If you want the sitemap, robots.txt and crawling of your own site checked together, I do that as part of a technical SEO audit.

Read next