Free AI Crawler Access Checker: GPTBot, ChatGPT and More

Free tool

Our server checks the public URL you enter. Results are not stored.

Loading the tool...

The AI Crawler Access Checker reads a site’s robots.txt and shows which AI crawlers are allowed or blocked. It covers 16 crawlers, including OAI-SearchBot and ChatGPT-User from OpenAI, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot. Enter a website address or paste a robots.txt, choose a path to test, and you get a status and the exact rule that applied for each crawler.

Last updated: 6 November 2026. Written by Saad Raza.

Key takeaways

ChatGPT uses three separate OpenAI crawlers. OAI-SearchBot finds pages for search results, ChatGPT-User fetches a page when a person asks for it, and GPTBot collects training data.
Blocking a training crawler does not block the search crawler, so you can opt out of training and still appear in AI search answers.
Robots.txt is a request, not a lock. A firewall, CDN rule or login can still stop a crawler that robots.txt allows.
A missing robots.txt means everything is allowed. A server error on robots.txt is treated differently, so fix it first.

What the AI Crawler Access Checker does

Many sites block AI crawlers by accident. A blanket rule copied from a template, a security plugin or an old training-data opt-out can also block the crawlers that find pages for AI search answers. This tool shows the result in one table.

The tool follows the robots.txt matching rules. It picks the group that names the crawler, or the * group if none does. It then applies the longest matching rule, and Allow wins a tie. Wildcards (*) and end anchors ($) are supported.

How to use the checker

Enter your website address and press the button. The tool fetches /robots.txt from your site. Or paste a robots.txt into the text box to test a draft before you publish it.
Leave the path as / to test the whole site, or enter a path such as /blog/ to test one section.
Read the four summary figures: crawlers checked, allowed, blocked and AI search bots blocked.
Read the table. Each row names the crawler, its company, its purpose, the status and the rule that decided it.
Fix any crawler you meant to allow, then run the check again on the updated file.

How to read the results

Purpose	Meaning	Usual choice
Search	Finds pages to cite in AI search answers	Allow if you want to be cited
User action	Fetches a page when a person asks the assistant to read it	Allow if you want people to use your pages in chats
Training	Collects content for model training	Your decision
Training control	A robots.txt token that controls training use, with no crawler of its own	Your decision

Example output

The Load example button tests this robots.txt: everything allowed for all crawlers, GPTBot and CCBot blocked from the whole site, and ClaudeBot blocked only from /private/. The result for the whole site is 16 crawlers checked, 14 allowed and 2 blocked. GPTBot and CCBot show as blocked. ClaudeBot shows as allowed, because the home page is not under /private/. All seven search and user-action crawlers are allowed.

The crawlers the tool checks

OpenAI: OAI-SearchBot, ChatGPT-User and GPTBot.
Anthropic: Claude-SearchBot, Claude-User and ClaudeBot.
Perplexity: PerplexityBot and Perplexity-User.
Google and Apple tokens: Google-Extended and Applebot-Extended control AI training use. They are tokens in robots.txt and not separate crawlers.
Search engines: Googlebot and Bingbot. Bing also feeds some Copilot answers.
Others: CCBot (Common Crawl), Bytespider, meta-externalagent and Amazonbot.

Crawler names change over time. Check each company’s documentation for the current list before you rely on a rule.

Common mistakes this finds

A global block: User-agent: * with Disallow: / blocks every crawler that has no group of its own.
Blocking the wrong bot: blocking GPTBot to opt out of training and thinking it hides the site from ChatGPT search. It does not, because OAI-SearchBot is separate.
Rules in the wrong group: a Disallow line placed under the wrong User-agent line.
A blocked robots.txt: a server error or login wall on /robots.txt itself.

Why AI crawler access matters for SEO

AI assistants cite pages they can fetch and read. If a search crawler is blocked, your pages are less likely to appear as sources in its answers. Letting search crawlers in, and keeping the page text readable without scripts, is the basic step before any other AI search work. Pair this check with the AI Search Readiness Checker and an llms.txt file.

Limits to know about

The tool reads robots.txt rules only. It cannot see firewall rules, bot-protection settings, rate limits or logins. It does not prove a crawler has visited your site. To see real visits, check your server logs. The tool also does not follow Crawl-delay or Sitemap lines.

Questions about the AI Crawler Access Checker

Does blocking GPTBot hide my site from ChatGPT?

No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages on request. Each follows its own robots.txt rules.

Should I block AI training crawlers?

That is your decision. Blocking them stops those crawlers from collecting your content for training. It does not remove you from AI search, provided the search crawlers stay allowed.

Why does the tool say a crawler is allowed when I think it is blocked?

The tool reads robots.txt only. A firewall, CDN or bot-protection setting can block a crawler at the server. Check your server logs and security settings.

What happens if my site has no robots.txt?

Crawlers treat a missing file as permission to crawl everything. A server error is different, so fix errors on the file itself.

Does the tool store the robots.txt I paste?

A pasted file is checked in your browser and not stored. When you enter a web address, our server fetches that site’s robots.txt once and does not store the result.

Is Google-Extended a crawler?

No. It is a robots.txt token that controls whether Google may use your content for Gemini training and grounding. Googlebot still crawls for Search.

If you want a full crawl and indexing review, my technical SEO service covers robots.txt, sitemaps and AI search access, and you can ask for a free audit through the contact page. You can also browse the other free technical SEO tools.