2

My robots:

User-agent: *
Disallow: /wp-admin
Disallow: /sistema
Disallow: /site
Disallow: /old

Sitemap: http://www.example.com.br/page-sitemap.xml

In the google test tool: https://www.google.com/webmasters/tools/robots-testing-tool is going normal: 'allowed'

But at ahref I have the feedback:

Fetching robots.txt took too long

I have already asked the hosting staff to see for us if there is any blocking of the crawler or the ips used by ahrefs and there is not.

I have already disabled all plugins on the site and nothing

Stephen Ostermiller
  • 99,822
  • 18
  • 143
  • 364
sNniffer
  • 121
  • 3

1 Answers1

0

I changed hosting and right after I tested the ahrefs crawler and everything worked perfectly, probably something from the host was blocking ahrefs ips.

Incredible is that the hosting staff did not know how to solve it.

sNniffer
  • 121
  • 3