How to Parse an XML Sitemap (Extract All URLs)

You need every product page on a shop with 4,000 products. You could write a crawler that clicks through every category and every “next page” link. Or you could read the list the shop has already published for Google: its sitemap.

What is a sitemap

A sitemap is an XML file that lists a site’s URLs, often with a note of when each page last changed. Search engines use it to find pages they might miss by following links.

It’s just as useful for scraping and SEO audits. Reading one file is faster than crawling a whole site, and it’s much kinder to the site’s server.

The XML sitemap format

Sitemaps follow the protocol documented at sitemaps.org. The root element is <urlset>, and inside it there’s one <url> element per page:

sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-02-01</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2026-01-15</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://example.com/blog/my-post</loc>
    <lastmod>2026-02-05</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.6</priority>
  </url>
</urlset>

Tag	Required	Meaning
`<loc>`	Yes	The full URL of the page
`<lastmod>`	No	When the page last changed, as a date such as `2026-02-01`
`<changefreq>`	No	How often it changes: always, hourly, daily, weekly, monthly, yearly or never
`<priority>`	No	How important it is compared with other pages on the same site, from 0.0 to 1.0 (default 0.5)

Sitemap index files

One sitemap can hold at most 50,000 URLs and be at most 50 MB before compression. Bigger sites split their URLs across several sitemaps and list those in a sitemap index, a sitemap of sitemaps:

sitemap_index.xml

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-posts.xml</loc>
    <lastmod>2026-02-07</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2026-02-06</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-categories.xml</loc>
    <lastmod>2026-01-30</lastmod>
  </sitemap>
</sitemapindex>

So before you parse anything, check the root element. A <urlset> lists pages. A <sitemapindex> lists more sitemaps, and you’ll need to fetch and parse each of those too.

Some sitemaps are also gzip-compressed and end in .xml.gz. Unzip those before parsing.

Finding a website’s sitemap

Start with robots.txt, the file at the root of a site that tells crawlers what they may visit. Sites can list their sitemaps there with a Sitemap: line, and many do:

https://example.com/robots.txt

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

No luck? Try the usual places:

Text

https://example.com/sitemap.xml
https://example.com/sitemap_index.xml
https://example.com/sitemap/sitemap.xml

WordPress, Shopify and Squarespace all generate sitemaps for you at predictable URLs, so on those sites one of these guesses usually works.

Why sitemaps matter for SEO

A sitemap helps search engines find new pages quickly. That matters most on large sites, where internal links don’t reach every page, and on new sites, which few other sites link to yet.

Search engines treat the extra tags as hints, and some hints get ignored. Google ignores <changefreq> and <priority> entirely. It does use <lastmod>, but only if the dates are consistently accurate. Setting every page to today’s date is the sitemap version of crying wolf.

How to use the sitemap parser

Open the sitemap parser. Enter the sitemap’s URL and select Fetch sitemap to load its XML into the editor, or paste the XML yourself.
Select Parse sitemap. You get every <loc>, with its last-modified date and priority when the sitemap has them, and a count of how many URLs were found.
If it’s a sitemap index, you get the list of child sitemaps instead. Fetch and parse each one to get its pages.
Select Copy all to copy the URLs, one per line, or Download to save them as sitemap-urls.txt.

Fetching only works for public URLs. If it fails, for example because the sitemap is gzipped or behind a login, download and unzip the file yourself and paste the XML in. The parsing itself happens in your browser.

The URL list makes a good starting point for an SEO audit, a look at a competitor’s site, or a scraping job.

How to parse an XML sitemap