2

I want to allow Google bots to crawl only my index.php and the content of some folders. Would that work?

User-agent: *

Allow: /index*.php
Allow: /folder1/
Allow: /folder2/
Allow: /folder3/
Allow: /folder4/

Disallow: /

The idea is for Google to crawl URLs like:

  • http://example.com/folder1/discussionA/topic/
  • http://example.com/folder1/discussionB/topic/
  • http://example.com/folder1/discussionC/topic/
  • http://example.com/folder2/discussionD/topic/

and exclude everything else.

Stephen Ostermiller
  • 99,822
  • 18
  • 143
  • 364
user3512251
  • 21
  • 1

1 Answers1

1

I generally focus on what I don't want them to crawl, but I think your sample seems on target. I would, however, remove the space between the first line and the second:

User-agent: *
Allow: /index*.php
Allow: /folder1/

Mozilla explains it in more detail

elbrant
  • 890
  • 1
  • 7
  • 16