X-Robots-Tag Explained: How to Noindex PDFs and Other Files With HTTP Headers

The X-Robots-Tag is an HTTP header that gives search engines indexing instructions such as noindex, the same way the robots meta tag does for web pages. Its main advantage is that it works on files that have no HTML head, including PDFs, images and documents, and it can be applied to many URLs at once through server rules.

What it looks like

X-Robots-Tag: noindex

You can combine directives, such as noindex, nofollow, and target specific crawlers. It is sent in the response header, so you will not see it in the page source.

When to use it

• Keep PDFs or downloads out of search results.
• Noindex image files or whole folders.
• Apply rules by file type across the site.
• Control indexing where you cannot edit the page template.

Crawlers must be able to see it

A search engine only sees the header if it can fetch the URL. If robots.txt blocks the file, the noindex is never read and the URL may still appear in results. This is the same logic as in noindex vs canonical vs robots.txt.

Setting it on a server

It is added through server configuration, such as Apache or Nginx rules, or a plugin or CDN setting. Match the rule carefully so you do not noindex more than intended.

How to check it

Enter the file address into my free HTTP header checker and look for the X-Robots-Tag line. You can also test a page in the URL Inspection tool.

Common mistakes

• A sitewide rule that accidentally noindexes everything.
• Leaving a staging header in place after launch.
• Conflicting instructions between header and meta tag.

Frequently asked questions

Next steps

If you suspect a header is hiding pages, contact me here for a technical check.

More in this series (HTTP header questions): start with Free HTTP Header Checker, then explore:

• Cache-Control Headers: What They Do and How They Affect SEO
• Security Headers and SEO: HSTS, CSP and the Headers Worth Setting
• Canonical in the HTTP Header: How to Canonicalise PDFs and Non-HTML Files
• Gzip and Brotli Compression: How to Check It Is Working