Open any WordPress site’s /sitemap.xml, and the file that comes back is usually correct and almost never thought about again. That is the trap. A sitemap is trivial to generate and easy to get subtly wrong, and the difference between a sitemap that helps and one that quietly wastes crawl attention lives entirely in the details: which URLs are in it, which fields you fill, and how you split the file when the site outgrows a single one.
A sitemap is an XML file listing the URLs a site wants search engines to discover, optionally with a small amount of metadata about each one. Google introduced its Sitemaps format in 2005, and the joint Sitemap Protocol with Yahoo and Microsoft followed in 2006, formalizing it as an open standard documented at sitemaps.org. The file lives at a known URL (commonly /sitemap.xml), gets referenced from robots.txt or submitted to search engines, and lists the URLs the site considers worth indexing.
The important framing: Google treats the sitemap as a hint, not a directive. Pages absent from the sitemap can still be indexed if found through links. Pages listed in the sitemap can still be ignored if Google decides they do not merit indexing. The sitemap is one of several discovery mechanisms, and for most pages on most sites, internal links and backlinks are the primary path. The sitemap earns its value on pages that are poorly linked internally, on large sites where crawl guidance matters, and for signaling “this URL exists” before Google has crawled it.
The XML format, and which fields Google actually reads #
A minimal sitemap wraps a list of URLs in XML:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2026-03-15</lastmod>
</url>
<url>
<loc>https://example.com/blog/url-structure</loc>
<lastmod>2026-02-20</lastmod>
</url>
</urlset>The only required element is <loc>, the URL itself. Every URL must be absolute, including protocol and domain; relative URLs are invalid, and the file must use UTF-8 encoding.
Three optional elements exist in the protocol: <lastmod> (last modification date), <changefreq> (how often the content changes), and <priority> (a relative importance score from 0.0 to 1.0). Here is where a lot of sitemap advice is out of date. Google has confirmed through its documentation and public statements from search representatives that it ignores both <changefreq> and <priority>. The metadata was abused for years (every site declaring every page priority 1.0 with changefreq “always”), so Google stopped weighing it. The values still validate against the schema, but they do not influence Google’s behavior. If a tutorial tells you to carefully tune priority per page, it is describing a knob that is not connected to anything.
<lastmod> is the one field that carries weight, and only when it is accurate. Google uses it to prioritize re-crawling pages that have actually changed, and Google’s own guidance is that the date should reflect a significant update to the page (main content, structured data, or links) rather than a trivial change like a copyright-year bump. Sites that stamp every URL with the current date on every sitemap regeneration eventually train Google to discount the field entirely. An inaccurate <lastmod> is worse than none, because it burns the one signal that works.
The sitemap index, for sites that outgrow one file #
A single sitemap file is capped at 50,000 URLs and 50 MB uncompressed. (Gzip compression reduces transfer size, but the uncompressed size is what counts against the limit.) Cross those thresholds and the file needs to be split.
A sitemap index solves this. The index is an XML file that lists other sitemap files; search engines fetch the index, then fetch each sitemap it references:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/sitemap-posts.xml</loc>
<lastmod>2026-03-15</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-products.xml</loc>
<lastmod>2026-03-15</lastmod>
</sitemap>
</sitemapindex>An index file carries the same limits: up to 50,000 sitemap entries and 50 MB, which lets a nested structure address millions of URLs. Almost no real-world site approaches that ceiling.
The strategic choice is how to split. Splitting by content type beats splitting by arbitrary file size. A site with 200,000 product URLs, 5,000 blog posts, and 500 static pages is better served by one sitemap per type than by chunks of 50,000 mixed URLs. The payoff shows up in Search Console: it reports indexing status per sitemap, so a drop in indexed products with steady indexed posts points straight at the products section. A single combined sitemap hides that differentiation, and each smaller file is easier to debug in isolation.
What to include, what to leave out #
The rule is short: the sitemap should list URLs that are valuable, canonical, and indexable. That is the same list of URLs the site actually wants to appear in search results.
Include the primary content without hesitation: homepage, category pages, product and service pages, blog posts. Include strategically the pages that are weakly linked from internal navigation and might otherwise be missed, plus recently published or significantly updated pages where the sitemap helps signal the change.
Exclude anything non-indexable. Non-canonical URLs (parameter variants, filtered versions, paginated component pages beyond the first, tracking-parameter URLs) should not appear. Neither should noindex pages, robots.txt-blocked pages (Google cannot crawl them anyway), or redirect sources (list the destination, not the source). Listing a non-indexable URL sends Google conflicting instructions (“index this” alongside “do not index this”) and dilutes the file’s value.
The common failure is neglect: a site removes a product, adds noindex, or sets up a redirect, but the sitemap generator never catches up, and Google keeps seeing sitemap URLs that conflict with page-level signals. The fix is regenerating after any change that affects indexability. For most CMS-driven sites this is automatic (WordPress with Yoast or RankMath, Shopify, Wix), and the automation usually excludes the right URLs. Manual review is for catching the cases it misses.
How Google finds and prioritizes the sitemap #
A sitemap does nothing until Google knows where it is. Three discovery paths cover the common cases. The most reliable is submission through Google Search Console, which queues the sitemap for processing and then reports status, error counts, and discovered-URL totals. Second is a reference in robots.txt, which any compliant crawler will pick up:
Sitemap: https://example.com/sitemap.xmlThe directive can appear anywhere in robots.txt, and multiple entries are allowed (though a sitemap index is cleaner than many robots.txt lines). Third is convention: crawlers often check /sitemap.xml and /sitemap_index.xml without being told. For Bing, Bing Webmaster Tools offers an equivalent submission interface worth using if Bing visibility matters.
Discovery is one-way. Telling Google about the sitemap does not force Google to crawl every URL in it; Google still decides based on its own priority calculations. Beyond discovery, the sitemap feeds crawl prioritization in a few ways. An accurate <lastmod> pushes changed URLs higher in the re-crawl queue. Inclusion itself is a weak canonicalization signal: Google’s documentation lists sitemap presence among the factors it weighs when choosing a canonical URL, ranked below redirects and below rel="canonical" annotations, but still contributing. And the sitemap directs, rather than expands, crawl budget: it does not grant a large site more crawling, but it points the existing budget at the URLs the site considers important. For small sites Google crawls everything regardless, so this matters mainly at tens or hundreds of thousands of URLs.
One limit worth stating plainly: the sitemap does not rescue an otherwise-undiscoverable page. A URL with no internal links and no backlinks may sit in the sitemap and still get deprioritized, because Google treats sitemap-only URLs with no supporting signals as low-confidence. The sitemap complements internal linking; it does not replace it.
Specialized sitemaps and international structure #
The protocol supports extensions for content that benefits from structured discovery, each with its own schema namespace. Image sitemaps use <image:image> tags to surface images (useful when images are the primary content or are loaded via JavaScript and missed by standard crawling). Video sitemaps use <video:video> with thumbnail, title, and description fields. News sitemaps use <news:news> for approved Google News publishers to flag recent articles for fast indexing. Each works alongside the standard URL sitemap rather than replacing it, and most sites only need the standard one.
International sites face a structural choice. The hreflang annotations that declare language and region targeting can live in the page HTML or in the sitemap, via the xhtml:link extension:
<url>
<loc>https://example.com/page</loc>
<xhtml:link rel="alternate" hreflang="en" href="https://example.com/page" />
<xhtml:link rel="alternate" hreflang="fr" href="https://example.com/fr/page" />
<xhtml:link rel="alternate" hreflang="de" href="https://example.com/de/page" />
</url>Each URL entry lists every language variant, and the same set repeats on each variant’s entry. The advantage is centralization: one sitemap update propagates the relationships across all pages, where the HTML approach requires editing every page’s header when a variant is added. The disadvantage is debuggability, since the relationships are invisible when inspecting a page directly. Google deprecated the dedicated International Targeting report in Search Console in September 2022; hreflang issues now surface in the Enhancements area and through tools like Screaming Frog’s hreflang report, the URL Inspection tool, and third-party validators. For small, simple language structures HTML is often easier; large multilingual catalogs benefit from sitemap centralization.
The anti-patterns worth auditing for #
Most bad sitemaps share the same handful of flaws, and an audit usually turns up several at once:
- Non-canonical URLs included. Variants, paginated component pages, and parameter URLs sit alongside their canonical versions. The generator should emit only canonical URLs.
- Noindex pages included. URLs marked noindex send conflicting signals. Filter out anything carrying a noindex meta tag or X-Robots-Tag header.
- Redirected URLs included. Old URLs that redirect still appear; the crawl follows the redirect but the listing is wasted. List only final destinations.
<lastmod>stamped to today on every regeneration. Every page looks freshly modified and Google discounts the signal. Update the date only on real content changes.- Size limits exceeded. A file over 50,000 URLs or 50 MB. Split by content type behind a sitemap index.
- Sitemap undeclared. It exists but is neither in robots.txt nor submitted to Search Console. Do both.
- Sitemap returns an error. The submitted URL 404s or has moved, and the Search Console error goes unnoticed. Verify it returns 200 with valid XML and monitor the report.
An eighth pattern hides in plain sight: stale sitemaps from a retired generator. Switch from Yoast to RankMath and the old sitemap URL often lingers alongside the new one, both submitted, one drifting out of date. Delete the old sitemap when you switch, and keep only the current one reachable.
FAQ #
Do I need a sitemap if my site is small and well-linked?
Not strictly. Google can discover a small, internally-linked site through crawling alone. A sitemap still helps by signaling changes via <lastmod> and giving Search Console a clean per-file indexing report, so the cost is near zero and the downside is none.
Will setting priority 1.0 on my key pages help them rank?
No. Google ignores the <priority> field entirely, so the value has no effect on crawling, indexing, or ranking. The same applies to <changefreq>.
Should I put every URL on my site in the sitemap?
Only the canonical, indexable ones you want in search results. Leaving in noindex pages, redirects, and non-canonical variants sends Google conflicting signals and dilutes the file.
How often should the sitemap update?
Whenever content that affects indexability changes: new pages, removed pages, new redirects, or noindex additions. Most CMS generators handle this automatically; the value is in the <lastmod> accuracy, not the frequency of regeneration.
One sitemap or a sitemap index?
Stay with one file until you approach 50,000 URLs or 50 MB, or until per-content-type indexing visibility becomes worth having. At that point, split by content type behind an index.
The sitemap is easiest to reason about not as a list of pages but as structured input to a system trying to crawl the site efficiently. Google needs to know what URLs exist, which to re-crawl first, how to spend a limited crawl budget, and which variant is canonical. Every design choice follows from those needs: accurate <lastmod> for re-crawl priority, content-type splitting for indexing visibility, and strict exclusion of non-indexable URLs to keep the input clean. Understood that way, the sitemap does its narrow job well and stops there. It will not grant crawl budget, override quality judgments, or rank a thin page. The sitemaps that pay off are the ones built by someone who knows exactly which of those things it can and cannot do.