
Short answer: To create a robots meta tag, add <meta name="robots" content="noindex, follow"> (with the rules you need) inside the <head> of the page, or set it per page in Yoast SEO or Rank Math. The page must stay crawlable, because Google ignores a noindex it cannot see when robots.txt blocks the URL. For PDFs and other non-HTML files, send the same rules in an X-Robots-Tag HTTP header.
This guide is for site owners, developers and SEOs who need to keep certain pages out of Google, control how pages appear in results, or work out why a page has vanished from search. You will be able to write the right tag for each situation, add it in plain HTML, WordPress, Shopify or a server header, check that Google sees it, and avoid the mistakes that quietly deindex good pages.
What the robots meta tag does
The robots meta tag is a page-level instruction to search engine crawlers. It sits in the HTML head and tells them whether to index the page, whether to follow its links, and how much of it to show in results. It does not stop crawling. That is the job of your robots.txt file, and confusing the two is the most common error I find on audits.
The tag has two attributes:
- name:
robotsaddresses all crawlers. A crawler’s own name, such asgooglebotorbingbot, addresses only that one. - content: one or more rules, separated by commas, such as
noindex, nofollow.
Google’s robots meta tag specification says the name, content and rule values are all case-insensitive, and that when rules conflict, the more restrictive one applies. With no tag at all, the default is index, follow, so you never need to add a tag just to say that.
The directives Google supports
| Rule | What it does | Typical use |
|---|---|---|
noindex | Keeps the page out of search results | Thank-you pages, internal search results, thin tag archives, staging |
nofollow | Asks Google not to follow links on the page | Rarely needed; user-generated pages you cannot moderate |
none | Same as noindex, nofollow | Pages you want ignored completely |
nosnippet | No text or video snippet in results; Google says this also applies to AI Overviews and AI Mode | Content you do not want quoted, such as paid material |
max-snippet:[n] | Limits the text snippet to n characters (0 = none, -1 = no limit) | Publishers controlling how much text is shown |
max-image-preview:[none|standard|large] | Sets the maximum image preview size | Most sites should use large for Discover eligibility |
max-video-preview:[n] | Limits video previews to n seconds | Video publishers |
noimageindex | Images on the page are not indexed | Photo libraries with licensing concerns |
notranslate | No translation link in results | Legal or medical text that must stay in its original language |
indexifembedded | Allows indexing when embedded in an iframe, used with noindex | Embeddable widgets and media players |
unavailable_after:[date] | Stops showing the page after a date | Event pages and time-limited offers |
Google lists all as the twelfth rule, which is simply the default. It also says noarchive, nocache and nositelinkssearchbox are no longer used and are ignored, so older guides that still recommend them are out of date. Other search engines support their own lists, so check each engine’s documentation if Bing or others matter to you.
For small pieces of text rather than a whole page, use the data-nosnippet attribute on a span, div or section, for example around a price that changes often.
Not sure if this is hurting your site?
Get the fixes that matter most, in priority order.
How to create a robots meta tag
In plain HTML
- Decide the rules (use the decision flow below if unsure).
- Open the page template and find the
<head>section. - Add one tag, for example
<meta name="robots" content="noindex, follow">. Use<meta name="googlebot" content="...">only when you need a Google-specific rule. - Publish, then check the live source (not your editor) to confirm there is only one robots tag.
If you would rather not hand-code it, my free meta tag generator builds the tag with the rest of your head tags.
In WordPress
- Yoast SEO: in the post editor open the Yoast panel, then Advanced. Set “Allow search engines to show this content in search results?” to No for noindex, and “Should search engines follow links on this content?” for nofollow. Whole content types and archives are set under Yoast SEO > Settings.
- Rank Math: open the Rank Math panel, then Advanced, and tick the Robots Meta options.
- Site-wide check: Settings > Reading > “Discourage search engines from indexing this site” adds noindex to every page. It is meant for development sites and is the first thing I check when a live WordPress site disappears.
In Shopify and other platforms
Shopify has no per-page robots setting in its default admin, so the usual approach is a conditional tag in theme.liquid that outputs noindex for specific templates or handles, or an SEO app that does the same. Wix and Squarespace both have a per-page “hide from search results” switch in their SEO settings, which adds noindex for you.
As an X-Robots-Tag header
PDFs, images and other files have no HTML head, so send the rules as an HTTP header instead. In Apache (.htaccess):
<Files ~ "\.pdf$"> Header set X-Robots-Tag "noindex" </Files>
In NGINX:
location ~* \.pdf$ { add_header X-Robots-Tag "noindex"; }
Headers are also useful for applying a rule to a whole folder or file type without editing templates. Confirm the header is actually sent with my HTTP header checker.
Which robots rule does this page need?
Should this page ever appear in Google’s results?
Do you want to limit how much of it Google shows (text, images or AI summaries)?
Is the concern the whole page, rather than one block such as a price or a quote?
Is it a duplicate or variant of another page you do want indexed (for example a filtered or tracking URL)?
Is it a PDF, image or other non-HTML file?
Result: no tag needed. The default is index, follow. If you only want to protect a small block, wrap it in a data-nosnippet attribute.
Result: use snippet controls. nosnippet removes text snippets entirely; max-snippet:[n] limits them. Keep max-image-preview:large unless you have a licensing reason not to.
Result: use a canonical tag, not noindex. Point the variant to the main URL with a canonical so signals are consolidated. Noindex the variant only if it has no main equivalent.
Result: noindex, follow in the HTML head. Keep the URL crawlable in robots.txt, and remove it from your XML sitemap.
Result: send an X-Robots-Tag header. Set X-Robots-Tag: noindex in the server configuration for that file or file type.
Robots meta tag vs robots.txt vs canonical
| Tool | Controls | Works when | Use it for |
|---|---|---|---|
| robots.txt | Crawling | Always read first | Stopping crawl waste on huge URL spaces (filters, internal search) |
| Robots meta tag | Indexing and how results look | Only if the page can be crawled | Keeping individual HTML pages out of results |
| X-Robots-Tag | Indexing and how results look | Only if the file can be crawled | PDFs, images, whole folders |
| Canonical tag | Which duplicate is shown | Treated as a strong hint | Variants of the same content; see canonical tags |
The key rule from Google’s noindex documentation: a page with noindex must not be blocked by robots.txt, or Google never sees the rule. Google’s robots.txt documentation adds that a blocked URL can still appear in results, without a description, if other sites link to it. Google also does not support noindex inside robots.txt. Test what your robots.txt actually blocks with my robots.txt tester.
A note on JavaScript
Google’s JavaScript SEO guidance says JavaScript can add or change a robots meta tag, but when Google meets noindex in the original HTML it may skip rendering, so using JavaScript to remove a noindex may not work. On React, Vue or headless sites, output the correct tag in the server-rendered HTML. My guide to JavaScript SEO covers rendering in more detail.
How to check the tag is working
- View the live source (Ctrl+U) and search for
name="robots". There should be one tag, not two from a theme and a plugin. - Check the headers for an X-Robots-Tag, which does not appear in the source.
- Use URL Inspection in Search Console. Under Page indexing, “Indexing allowed?” shows whether Google found a noindex. If you have not set up Search Console yet, start with adding your site to Search Console.
- Watch the Page indexing report for “Excluded by ‘noindex’ tag” and make sure every URL listed there is one you meant to exclude.
Mistakes I find on audits
- Noindex left on after launch. A developer ticks “Discourage search engines” on staging, the site goes live, and traffic never arrives.
- Noindex plus a robots.txt block on the same URLs, so Google cannot see the noindex and the pages linger in results.
- Noindexed URLs in the XML sitemap. It sends Google mixed signals; XML sitemaps should list only pages you want indexed.
- Two robots tags, one from the theme and one from a plugin, with different rules. The more restrictive one wins.
- Noindex on paginated categories on ecommerce sites, which can cut off products linked only from page 2 onwards. See pagination in SEO.
- Using noindex to fix duplicate content that a canonical or redirect would handle better.
Frequently asked questions
Where do I put the robots meta tag?
Inside the <head> section of the HTML page, before </head>. In WordPress, let Yoast SEO or Rank Math output it instead of editing theme files.
What is the difference between noindex and nofollow?
Noindex keeps the page out of search results. Nofollow asks search engines not to follow the links on the page. You can use either alone or both together; noindex, follow is the most common combination for pages you want hidden but still crawled through.
How long does noindex take to remove a page from Google?
Google has to recrawl the page first. Google says that can take months for less important pages. Requesting indexing in URL Inspection after adding the tag usually speeds it up, and the Removals tool hides a URL temporarily while you wait.
Can I use noindex in robots.txt?
No. Google does not support a noindex rule in robots.txt. Use the robots meta tag or the X-Robots-Tag header, and keep the URL crawlable so Google can read it.
Does nosnippet stop my content appearing in AI Overviews?
Google’s specification says nosnippet also applies to AI Overviews and AI Mode. It also removes your normal text snippet, which can reduce clicks, so use it only on pages where that trade-off makes sense.
Is the robots meta tag a ranking factor?
No. It controls whether and how a page appears, not how well it ranks. Used well, it keeps low-value pages out of the index so Google’s attention goes to the pages that matter.
Sources
- Google Search Central, “Robots meta tag, data-nosnippet, and X-Robots-Tag specifications” (accessed 2026).
- Google Search Central, “Block Search indexing with noindex” (accessed 2026).
- Google Search Central, “Understand JavaScript SEO basics” (accessed 2026).
- HTTP Archive, “Web Almanac 2024: SEO chapter” (2024).
If pages have dropped out of Google and you suspect a stray noindex, a robots.txt conflict or a plugin clash, my technical SEO services include a full indexing review, or you can describe the problem to me.
Request received
Thank you! Saad will contact you within the next 24 hours.
Keep an eye on your phone and inbox.
