XML Sitemaps are files that list all the important URLs on a website so that search engine crawlers can efficiently and fully crawl a website.
XML Sitemaps are files that list all the important URLs on a website so that search engine crawlers can efficiently and fully crawl a website. The specification for them lives on sitemaps.org.
An example of a simple XML sitemap is:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://www.example.com/</loc></url>
<url><loc>https://www.example.com/page.html</loc></url>
<url><loc>https://www.example.com/another-page.html</loc></url>
</urlset>
The sitemap for a site is typically named "sitemap.xml" and is served from the root directory of the site (https://www.example.com/sitemap.xml). However, that is just a convention and sitemaps can have any file name. See Does name of sitemap file that Wordpress generates matter? Some bots may look for the sitemap at its typical name and find it automatically. If a sitemap has a different name, it will need to be submitted to search engines through webmaster tools, or by pointing to it from robots.txt like:
User-agent: *
Disallow:
Sitemap: http://www.example.com/example-sitemap.xml
Sitmaps can be compressed using gzip. Because they are often very large, it is very common to find sites serving compressed sitemaps like sitemap.xml.gz.
In addition to the URLs, other data can be included in XML sitemaps: last modified dates (lastmod), change frequency (changefreq), priority, alternate language URLs (hreflang), video URLs, and image URLs. All of those extra fields are optional. Some of them (like "last modified" and "change frequency") are never worth including because Google says it doesn't use them. See How important is it to include <lastmod> in a sitemap?
Sitemaps are limited to 50,000 URLs and 50MB. Sites with more URLs than that will need to use multiple sitemaps. Those sitemaps can be submitted to search engines individually or included in one "sitemap index" file which is submitted to search engines. See Google Sitemap Limits?
Despite broad search engine support, sitemaps have surprisingly little impact on SEO. Search engines don't promise to index every URL in a sitemap, and in fact usually won't index a URL which can be found is through a sitemap. Sitemaps don't help with search engine rankings either. At best, sitemaps get a site fully crawled by search engine bots, give extra stats in webmaster tools, and tell search engines about your preferred URLs. See The Sitemap Paradox.
The best way to generate a sitemap is to use a program to list all the URLs of the website from the file system or database. Popular content management systems (like Drupal, and WordPress) have plugins that can do just that. Generating a sitemap by crawling your website is not recommended. If crawling your site can list all the pages, search engines will be able to crawl the whole thing with or without the sitemap. Sitemaps are typically regenerated automatically on a daily or weekly basis by an automated process scheduled to run on the web server.
When generating a sitemap, all the URLs on the site that have content should be included. Search engines don't want sitemaps to include error pages, alternate URLs, or redirects.
