The X-Robots-Tag HTTP header can be used to send directives to Google and other search engines. See this page for a discussion from Google:
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
In Apache you can implement the X-Robots-Tag using the Header directive. For example the following code will mark PDF documents as “noindex, nofollow”:
|
1 2 3 4 5 |
<Files ~ "\.pdf$"> Header set X-Robots-Tag "noindex, nofollow" </Files> |
