Use noindex to keep a page out of search results, a canonical tag to say which of several similar URLs should be the main one, and robots.txt to stop crawlers requesting URLs at all. They are not interchangeable, and combining them wrongly, such as blocking a noindex page in robots.txt, makes the instruction invisible.
In the HTTP Archive Web Almanac 2024, canonical tags appeared on 65% of mobile pages and 69% of desktop pages. These signals are everywhere, which is why mixing them up is one of the most common technical SEO faults.
Three tools, three different jobs
People reach for these three tools as if they were three ways of saying “do not show this page”. They are not. Each one acts at a different stage of how a search engine handles a URL, and each answers a different question.
- Robots.txt answers: may a crawler request this URL at all? It controls crawling.
- Noindex answers: may this page appear in search results? It controls indexing, and it only works if the crawler can read the page.
- Canonical answers: of several URLs with the same or very similar content, which one should represent them? It controls consolidation.
When I audit a site, a large share of indexing problems turn out to be one of these tools being used for another tool’s job. A site owner wants a page out of Google, so they disallow it in robots.txt. The page then stays in the index as a bare URL, and nobody can work out why. The rest of this guide explains what each tool does, what it does not do, and how to pick between them.
Robots.txt: controlling crawling, not indexing
Google’s introduction to robots.txt says the file tells crawlers which URLs they can access, and is used mainly to avoid overloading your site with requests. It also says robots.txt is not a mechanism for keeping a page out of Google, and that a disallowed page can still be indexed if other sites link to it. In that case the URL may appear in results without a description, because the crawler never read the page.
So robots.txt is the right choice when the problem is crawling: endless filter combinations, internal search results, or admin areas that waste server capacity. It is the wrong choice when the problem is visibility. I go through the syntax and common patterns in my guide to robots.txt.
Two more limits are worth remembering. The rules are not enforced, so a crawler can choose to ignore them. And robots.txt is public, so it should never be used to hide sensitive pages, since it lists their paths for anyone who looks.
Not sure if this is hurting your site?
Get the fixes that matter most, in priority order.
Noindex: keeping a crawlable page out of results
Noindex is a rule delivered by a robots meta tag in the HTML, or by an X-Robots-Tag HTTP header, which is how you do it for PDFs and images. Google’s documentation on blocking indexing describes noindex as a rule used to prevent indexing content by search engines that support it, Google included.
Noindex fits pages that must exist for users but have no reason to rank: thank you pages, internal search results, login pages, thin tag archives, and staging copies. The page stays visible to visitors and gets dropped from results once Google recrawls it and sees the rule. The tag goes in the head of the page, or in the response header for non-HTML files.
The critical condition is in Google’s own wording. For noindex to work, the page must not be blocked by robots.txt and must be otherwise accessible to the crawler. If it is blocked, the crawler never sees the noindex rule, and the page can still appear in search results.
Canonical: choosing one URL among duplicates
A canonical tag is a link element in the head of a page that names the preferred URL for that content. It is meant for duplication and near-duplication: the same product under several category paths, a page reachable with and without tracking parameters, or a syndicated article. Google’s page on consolidating duplicate URLs describes a rel=”canonical” annotation as a strong signal that the specified URL should become canonical, a redirect as a strong signal too, and sitemap inclusion as a weak one.
The word to hold on to is signal. A canonical is a strong hint, not a command, and Google can choose a different canonical if other signals disagree. If you want the full explanation, I cover it in what a canonical tag is.
Canonical is the right tool when you want the duplicate versions to remain accessible but consolidated. It is the wrong tool for removing a page from results, because the canonical page is the one that will show, and the duplicates simply fall away. Parameter-heavy sites meet this daily, which I explain in URL parameters and SEO.
Side by side: what each one actually does
| Robots.txt | Noindex | Canonical | |
|---|---|---|---|
| Main purpose | Control crawling | Keep a page out of results | Consolidate duplicates |
| Where it lives | One file at the site root | Meta tag or HTTP header per page | Link tag or header per page |
| Needs the page to be crawled? | No, it stops crawling | Yes, the page must be readable | Yes, the page must be readable |
| Strength | A rule for compliant crawlers | A rule Google follows once it sees it | A strong signal, not a command |
| Page stays accessible to users? | Yes | Yes | Yes |
| Can the URL still appear in results? | Yes, if linked elsewhere | No, once processed | Possibly, if Google picks it |
| Passes signals to another URL? | No | No | Yes, that is the point |
Combinations that conflict
Most of the damage I see comes from stacking these tools on the same URL. Here are the combinations to avoid and what to do instead.
Noindex plus a robots.txt block
This is the classic mistake. The robots.txt rule stops Googlebot from fetching the page, so it never reads the noindex, and the URL can stay in results. If you want a page out of the index, allow crawling first, let Google see the noindex, and only block crawling later if you still need to, after the page has dropped out.
Noindex plus a canonical to another page
These send opposite messages. A canonical says “treat this as a copy of that page”, while noindex says “do not index this one”. Google advises against using noindex to prevent canonical selection within a site, because it completely blocks the page from search. Choose one: if the page is a duplicate, canonicalise it, and if it should not be found at all, noindex it.
Canonical pointing at a blocked, redirected or noindex page
If the canonical target is blocked from crawling, redirects elsewhere, or carries a noindex, the signal points at something Google cannot use. The canonical should name a live page that returns a 200 and that you want indexed. The redirect side of this is covered in redirect chains and loops.
Robots.txt as a canonicalisation tool
Google says explicitly not to use robots.txt for canonicalisation. Blocking a duplicate stops Google reading its canonical tag and merging its signals, which defeats the purpose.
Which to use: a decision table by scenario
| Scenario | Use | Why |
|---|---|---|
| Thank you or confirmation page | Noindex | Useful to visitors, useless in results |
| Internal site search results | Noindex, or robots.txt if they waste crawling | Thin and endless |
| Same product in several categories | Canonical to the main URL | Duplicate content to consolidate |
| Tracking parameters on a page | Canonical to the clean URL | Same page, different address |
| Infinite filter combinations | Robots.txt for crawl control, canonical or links designed carefully | The issue is crawl volume |
| Staging site | Password protection, plus noindex as a backup | Real access control beats a signal |
| Admin and login areas | Authentication, and robots.txt to cut crawl waste | Not for public search |
| Page deleted permanently | 404 or 410, or a 301 to a close match | None of these three tools is needed |
| PDF you want out of results | X-Robots-Tag noindex | There is no HTML head to carry a meta tag |
| Old page replaced by a new one | 301 redirect | A redirect consolidates more clearly than a tag |
The last two rows matter. Often the right answer is none of the three tools. If a page is gone, the correct response is a status code, and I explain which one in HTTP status codes for SEO.
How to check what a URL is actually telling Google
- Open the page source or the response headers and look for a robots meta tag, an X-Robots-Tag header and a canonical link. Check the rendered HTML too if your site relies on JavaScript.
- Test the URL against your robots.txt file, to see whether Googlebot is allowed to fetch it.
- Use the URL Inspection tool in Search Console to see whether the URL is indexed, which canonical you declared and which one Google selected.
- Crawl the site and filter for pages that are noindex and disallowed, canonicalised to a non-200 page, or canonicalised to a different URL than themselves.
- Check the XML sitemap. It should list only canonical, indexable 200 URLs, because a sitemap full of noindex or redirected URLs contradicts your other signals.
If these tools are applied across a whole WordPress install through a plugin, review the global settings too. Many sites noindex entire archive types by default, and some plugins add canonicals automatically. My WordPress technical SEO checklist lists the defaults I check first.
Questions about noindex, canonical and robots.txt
Can I use robots.txt to remove a page from Google?
No. Robots.txt controls crawling, and a disallowed page can still appear in results if other pages link to it. To remove a page, let Google crawl it and add a noindex rule, or protect it with a password.
Is a canonical tag the same as noindex?
No. A canonical consolidates similar pages onto a preferred URL, and the preferred URL stays indexable. Noindex removes the page itself from results. Using both on one URL sends conflicting messages.
Does noindex stop Google from crawling a page?
No, it needs crawling to work. Google must fetch the page to see the rule. Noindex pages are still requested, so use robots.txt if the aim is to cut crawl volume.
Which should I use for duplicate content?
Use a canonical tag or a 301 redirect, depending on whether the duplicates need to stay accessible. A redirect is the clearer choice when the extra URLs have no reason to exist.
These three controls look small, but a single wrong rule can hide a whole section of a site. If you are not sure what your pages are telling Google, my technical SEO services include a full review of crawl and index controls, and you can request a free audit to get started.
More in this series (Technical SEO in depth): start with WordPress Technical SEO Checklist: What to Fix First, then explore:
- HTTP Status Codes for SEO: 200, 301, 302, 404, 410 and 5xx
- Redirect Chains and Loops: How to Find and Fix Them
Request received
Thank you! Saad will contact you within the next 24 hours.
Keep an eye on your phone and inbox.