How to Test Robots.txt in Google Search Console

Google retired its old robots.txt Tester, so you now check your file with two things in Search Console: the robots.txt report, which shows whether Google can fetch your file, and the URL Inspection tool, which tells you whether a specific URL is blocked. Together they show what Google actually sees.

Use the robots.txt report

Open Settings in Search Console and look for the robots.txt report. It lists the robots.txt files Google found for your property, when they were last fetched, their status and any problems. If Google cannot fetch the file, the report is where you find out.

Test an individual URL

1. Paste the page address into the inspection bar at the top.
2. Open the result and view the crawl details.
3. Check whether crawling is allowed or blocked by robots.txt.

My URL Inspection tool guide walks through the screens.

Test before you publish a change

Search Console only reflects the live file, so check your draft rules elsewhere first. Paste a URL into my robots.txt tester and AI crawler checker to see which crawlers are allowed or blocked.

Ask Google to recrawl

After you fix or update the file, use the report’s option to request a recrawl so Google picks up the new version sooner. Changes are normally picked up within a day or so, but timing varies.

What to look for

• A fetch status that is not successful.
• A file that is much larger than expected.
• Rules that block pages you want in search, as covered in indexed though blocked by robots.txt.

For wider indexing problems, see the Search Console coverage report explained.

Frequently asked questions

Next steps

If you are unsure what your robots.txt is doing, contact me here for a technical check.

More in this series (robots.txt testing questions): start with Robots.txt Tester and AI Crawler Checker, then explore:

• Robots.txt Returns 404 or 500: What Google Does and How to Fix It
• How to Block Images From Google With Robots.txt
• Robots.txt User-Agent Groups: Which Rules Does Googlebot Follow?
• Does Robots.txt Apply to Subdomains and HTTP vs HTTPS?