Why important/big sites don't include a sitemap?

Question

I've been looking at most important sites and none of them have a sitemap.xml defined. My list includes:

• Amazon
• Ebay
• Yahoo
• Microsoft
• ... and others

But none of them include a sitemap.xml file. Why is this? It's not that important?

Note: I've been using the Google Bot user agent to check it. The only site that returned a 200 Ok was google.com

Answer 1

Just checking sitemap.xml is not enough. There are several reasons that that check might fail even though the site has a sitemap.

1. Did you check sitemap.xml.gz? Google supports gzip of sitemaps. Large sites with large sitemaps are likely to take advantage of this feature.
2. You can specify the name of the sitemap file in robots.txt. It doesn't have to be sitemap.xml. A robots.txt can have a line like this that says where to get the sitemap. Sitemap: http://example.com/mysitemapfile.xml
3. It is also possible to create a sitemap file at a "secret" URL. You can tell Google about this URL through webmaster tools. That way Google will get your sitemap, but it won't be easily accessible to the public.

---

Looking in robots.txt, many of the sites you list do have sitemaps.

Amazon

Sitemap: http://www.amazon.com/sitemap-manual-index.xml
Sitemap: http://www.amazon.com/sitemap_vendor_videos_us.xml
Sitemap: http://www.amazon.com/sitemap_vod_index.xml
Sitemap: http://www.amazon.com/sitemaps.f3053414d236e84.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.1946f6b8171de60.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.bbb7d657c7e29fa.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.11aafed315ee654.SitemapIndex_0.xml.gz
Sitemap: http://www.amazon.com/sitemaps.c21f969b5f03d33.SitemapIndex_0.xml.gz

Ebay

Sitemap: http://www.ebay.com/lst/PDP_US_main_index.xml
Sitemap: http://www.ebay.com/lst/PDP_US_incremental_index.xml
Sitemap: http://www.ebay.com/lst/VIP_US_index.xml
Sitemap: http://www.ebay.com/lst/SRP_US_index.xml

Microsoft

...
Sitemap: http://www.microsoft.com/en-us/dynamics/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/kinectforwindows/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/sqlazure/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/cloud/sitemap.xml
Sitemap: http://www.microsoft.com/en-us/server-cloud/sitemap.xml
...