5

Google is discovering some URLs in GWT which I'm surprised it could discover. The only place these are linked from are as follows:

<li class="multiselect__item" data-key="filter__brand--examplebrand2" data-name="examplebrand2" data-count="92" data-url="/example-category/examplebrand1--examplebrand2/" title="examplebrand2">    <input type="checkbox" id="checkboxfilter__brand--examplebrand2">    <label for="checkboxfilter__brand--examplebrand2"><span></span>examplebrand2<i class="multiselect__item-count"> (92)</i></label></li>

The code below being the interesting bit:

data-url="/example-category/examplebrand1--examplebrand2/"

GWT is reporting that these URLs are linked from a page on the site, and not externally. My question, should Google be able to crawl these URLs?

Stephen Ostermiller
  • 99,822
  • 18
  • 143
  • 364
Max
  • 6,146
  • 18
  • 40

1 Answers1

6

Googlebot uses heuristics to pick out anything from the page that looks like it might be a URL. It can crawl:

  • URLs in SELECT option values
  • URLs in data attributes
  • URLs in JavaScript variables

In my experience, just having a slash (/) in a string is enough for Googlbot to think it may be a URL. The fact that you name you attribute data-URL may also contribute to Googlebot recognizing it.

As far as I know, Googlebot only uses these "links" for content discovery. I don't believe that it passes link juice across them the same way that it does for normal a href links.

Unfortunately, if Googlebot gets it "wrong" and crawls something that isn't a URL, it will still report it in your site errors in Google Webmaster Tools. In my opinion, it shouldn't do that when it used a heuristic to try to find something that looks like a URL.

Stephen Ostermiller
  • 99,822
  • 18
  • 143
  • 364