2

I have been using Raven Tools, and I want these dynamically generated thumbnails to be not crawled or not be flagged as No TITLE images.

I tried adding X-Robots-Tag "noindex, nofollow" on each image within my webserver (nginx), but Raven's Site Auditor still reads the images and flags them.

How do I prevent them from being crawled?

Edit: The problem is Raven Tools flag these dynamically generated images as NO TITLE, since a WP plug-in generates its files.

Batutoy Tria
  • 21
  • 2

1 Answers1

1

This is not a direct answer, but you should cache your dynamically generated thumbnails so they are generated only once, and then served only from the cache.

Generating thumbnails is a very expensive process (for server CPU and memory). So you really should avoid doing it all the times.

My suggestion then is:

  • When user requests a page that has thumbnails, page should check if the cached thumbnail image is present.
  • If a cached thumbnail do not exists yet, then the page must generate the thumbnail and save it to the cached thumbnails directory.
  • The page will always output the cached image URL, and never a URL for a thumbnail-generating script.
  • You should generate all thumbnails in a specific directory on your server (like /thumbs/), so you can add this directory to robots.txt and this way you can "ask" for the crawler to not index images on this folder.
Daniel Ribeiro
  • 151
  • 4