10

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt.

Disallow: /page-one.html
Noindex: /page-two.html

Seems like it would prevent search engines from crawling page one, and prevent them from indexing page two.

Is this robots.txt directive supported by Google and other search engines? Does it work? Is it documented?

Stephen Ostermiller
  • 99,822
  • 18
  • 143
  • 364

1 Answers1

9

Google used to unofficially support a Noindex directive in robots.txt, however in 2019, they announced that the directive will no longer work.

Here is what Google's John Mueller says about Noindex: in robots.txt:

We used to support the no-index directive in robots.txt as an experimental feature. But it's something that I wouldn't rely on. And I don't think other search engines are using that at all.

Before Google announced the feature was discontinued, deepcrawl.com did some testing of the feature and discovered that:

  • Before 2019, it still worked with Google
  • It prevented URLs from appearing in the search index
  • URLs that have been noindexed in robots.txt were marked as such in Google Search Console

Given that Google discontinued the feature, it shouldn't be used anymore.

Instead, use robots meta tags that are well supported and documented to prevent indexing:

<meta name="robots" content="noindex" />
Stephen Ostermiller
  • 99,822
  • 18
  • 143
  • 364