Indexed, Though Blocked by Robots.txt: Why It Happens and How to Fix It

Indexed, though blocked by robots.txt means Google found the URL, usually through links, and added it to its index, but could not crawl it because robots.txt blocks it. Robots.txt controls crawling, not indexing. To keep the page out of search you must let Google crawl it and add a noindex instruction, or remove the page.

Why this happens

Google can learn a URL exists from internal or external links. If it cannot fetch the page, it cannot read any noindex tag on it, so the URL may still appear in results with little or no description. That is exactly the situation this warning describes.

Blocked vs indexed though blocked

Search Console separates two statuses. Blocked by robots.txt means the URL is not indexed. Indexed, though blocked by robots.txt means it is indexed anyway. Both start with a rule in your robots.txt, but the fix depends on what you want.

If you want the page indexed

1. Find the Disallow rule that matches the URL.
2. Remove or narrow that rule.
3. Test the URL in the robots.txt tester.
4. Request a recheck with the steps in my URL Inspection tool guide.

If you want the page removed from search

1. Remove the Disallow rule so Google can fetch the page.
2. Add a noindex robots meta tag or an X-Robots-Tag header to the page.
3. Wait for Google to recrawl and drop it.
4. Only then, if you wish, consider blocking crawling again.

For urgent cases, use the removal tool in Search Console for a temporary hide while the noindex takes effect. For a full comparison of the options, read noindex vs canonical vs robots.txt.

Common causes

• A staging rule left on the live site.
• Overly broad wildcard patterns.
• Blocking a folder that contains pages you wanted indexed.
• A CMS setting that adds rules automatically.

Rebuild a safe file using my robots.txt generator.

Frequently asked questions

Next steps

If you have many affected URLs and are unsure which to keep, contact me here for an indexing review.

More in this series (robots.txt questions): start with Free Robots.txt Generator, then explore:

• How to Add a Sitemap to Robots.txt
• Robots.txt Wildcards: How * and $ Work With Examples
• WordPress Robots.txt: Where It Is, How to Edit It and What to Include
• Does Google Respect Crawl-Delay in Robots.txt?