Webmasters Stack Exchange
Webmasters Stack Exchange

Questions tagged [googlebot]

Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.

Googlebot is the search bot software used by Google, which collects documents from the web to build a searchable index for the Google Search engine. Googlebot is also know as a robot and more suitable questions and answers may be found under

944 questions
41
votes
2 answers

What is .well-known/traffic-advice directory?

I'm getting lots of requests from Googlebot on the .well-known/traffic-advice directory on my server. What is it and what are they looking for? Should I add something to this directory, and if yes what?
WendiT
  • 543
  • 1
  • 4
  • 5
30
votes
4 answers

Prevent XML sitemaps from showing up in Google search results

How do I prevent my XML sitemap files from showing up in Google search results like this result of a site: search query: I don't understand why Google would choose to show sitemap files in search results to begin with. These files are not meant…
Stephen Ostermiller
  • 99,822
  • 18
  • 143
  • 364
28
votes
1 answer

Why does my IP address become Google's IP address when using Chrome on the mobile?

I am tracking every member's geolocation (using db-ip) and everything is fine except mobile phones with the Chrome browser. They always give me the result US Mountain View California ip:66.249.xxx.xxx. This is Google; I am 100% sure about it. But…
ozgur
  • 475
  • 4
  • 8
25
votes
3 answers

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO and spiders navigating the website do I get…
Howdy_McGee
  • 642
  • 1
  • 6
  • 15
19
votes
3 answers

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that Baiduspider crawls at about the same frequency as…
samthebrand
  • 910
  • 1
  • 10
  • 30
17
votes
7 answers

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented Googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact URLs. Does Google cache robots.txt and, if so,…
DDM
  • 341
  • 1
  • 2
  • 6
14
votes
1 answer

Should I return a 429 or 503 status code to a bot?

We are doing some work to block excessive bots. We are throttling requests based on IP. In the past, using IIS, we have returned a 503 error to the client. Someone suggested that a 429 is more appropriate. I am trying to figure out which one I…
Evik James
  • 685
  • 5
  • 14
13
votes
3 answers

Can you use googleon and googleoff comments to prevent Googlebot from indexing part of a page?

I've seen code like for preventing Google from indexing part of a page:

This is a paragraph that will be indexed by Google.

This is a paragraph that will NOT be indexed by Google.