To test robots.txt, enter a page URL into the free Robots.txt Tester and AI Crawler Checker above. It shows whether Googlebot, Bingbot, GPTBot, ClaudeBot or PerplexityBot may crawl that URL under the robots.txt standard, and it also checks whether the site has an llms.txt file.
Google explains that if a page is disallowed from crawling through robots.txt, any information about indexing or serving rules will not be found and will therefore be ignored, so a blocked page cannot carry a working noindex.
How to use the Robots.txt Tester
- Enter the full address of the page you want to test, such as a product page, a blog post or a folder.
- Run the test. The tool fetches the robots.txt file from that site and applies its rules to your URL.
- Look at the answer for each crawler: Googlebot, Bingbot, GPTBot, ClaudeBot and PerplexityBot.
- Check the llms.txt result, which tells you whether that file exists on the site.
- Change the rules in your robots.txt if the answer is not what you intended, then test again.
How to read the results
The key output is an allowed or blocked answer per crawler for the URL you entered. Crawlers can have different answers, because a file may give one bot its own group of rules and leave others on the general rules.
- Allowed means that crawler is permitted to fetch the URL under the rules in the file.
- Blocked means a rule disallows it. Find the matching line in your file to see why.
- Different answers for different bots usually means there are named groups, for example one for GPTBot. That is fine if you chose it.
The llms.txt check simply reports whether the file is present. It is an emerging convention, not a ranking factor, and the tool does not judge its contents.
A good result is one where every page you want found is allowed for the crawlers you care about, and every area you deliberately closed is blocked.
Common problems this finds
- Important pages blocked by accident. A broad rule such as a blocked folder can catch pages you want indexed. Narrow the rule or add a more specific
Allowline. - Staging rules on a live site. A leftover blanket block hides everything. Remove it and test the home page again.
- AI crawler blocks you did not choose. Some themes, plugins and hosts add rules for AI crawlers. If a result shows blocked for GPTBot or ClaudeBot and you did not intend it, trace the rule and remove it.
- No robots.txt where you expected one. If the file is missing, crawlers generally treat the whole site as open. Create one if you want to give guidance.
- Noindex on a blocked page. The tag will never be read. Either unblock the URL so the directive can be seen, or accept that it may still appear in results without a snippet.
A worked example
This is an illustrative example. A clinic adds a rule to block its /booking/ folder. A month later its new /booking-guide/ article is not being picked up. You test the article URL and Googlebot shows as blocked. The reason is a rule written without a closing slash, which matches any path starting with those letters. Changing the rule to /booking/ fixes it, and a second test shows the article allowed.
Limits of this test
The test runs from this site’s server and reads the public robots.txt of the site you enter, so it only works for sites that are reachable and do not block automated requests. It tells you what the file permits, not whether a crawler will actually visit, and it cannot see rules a crawler may apply on its own. Robots.txt is a voluntary standard, so a bot that does not follow it is not stopped by it. There is a fair-use limit on repeated checks. A blocked result for a crawler does not mean a page is gone from search, and an allowed one does not mean it will be indexed.
To write a new file, use the Robots.txt Generator. To check the headers that may also carry indexing rules, use the HTTP Header Checker.
Questions about the Robots.txt Tester and AI Crawler Checker
Which crawlers can I test?
You can check Googlebot, Bingbot, GPTBot, ClaudeBot and PerplexityBot against a URL. The answer for each follows the robots.txt standard, so named rules for a crawler take effect for that crawler.
Does blocking GPTBot or ClaudeBot affect my Google rankings?
Those rules apply to the named crawlers, not to Googlebot. Googlebot has its own answer in the results. Whether to allow AI crawlers is a business decision about how you want your content used.
What is llms.txt and do I need one?
It is a plain file some sites publish at the root to give language models a guide to their content. The tool only reports whether it exists. Nothing here suggests you must have one to rank.
Back to all free technical SEO tools or the full toolkit. If you would rather have someone check the whole site, see my technical SEO services or ask for a free audit.