AI Crawler Checker
See which AI bots — GPTBot, ClaudeBot, PerplexityBot and 19 more — can crawl your site.
What is an AI crawler checker?
An AI crawler checker tells you which AI bots can reach your site and which are blocked. Enter a domain, and optionally a specific path, and the tool fetches your live robots.txt, then evaluates it for around 22 AI crawlers the same way a compliant bot would. Each gets an allowed or blocked verdict plus the rule that decided it.
The crawlers are grouped by what they do, because that matters more than the brand name. Training crawlers such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-training-data-crawler and Amazonbot collect content that may be used to train models. AI search crawlers such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot index pages so assistants can cite them in answers. Googlebot and Bingbot are included in this group because Google AI Overviews and Microsoft Copilot draw on their indexes. User-triggered fetchers such as ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User and Meta-ExternalFetcher load a page only when a person asks an assistant to read it.
robots.txt is only one layer, so the checker looks further. It reports the HTTP status of robots.txt, reads X-Robots-Tag headers and meta robots tags on the page, runs a user-agent firewall test against your CDN or WAF, and checks whether you publish an /llms.txt file.
How to check AI crawler access
- 1
Enter your domain and path
Type the domain, for example example.com. Add a path such as /blog/ to test a specific section, since many files block deeper folders only.
- 2
Run the check
The tool fetches robots.txt server-side, evaluates each AI crawler against your path and requests the page to read headers and meta tags.
- 3
Read the verdict for each group
Scan the training, AI search and user-triggered groups separately. For any bot, open the details to see which user-agent group matched and which Allow or Disallow line produced the result.
- 4
Review the firewall test
Compare the HTTP status returned to GPTBot, ClaudeBot and PerplexityBot user agents with the status returned to a normal browser. A 403 or challenge page for bots only means your CDN or WAF is blocking them, regardless of robots.txt.
- 5
Test changes in paste mode
Switch to paste mode, drop in a draft robots.txt and rerun the evaluation. Confirm every result before deploying.
When to use an AI crawler checker
Auditing a robots.txt you inherited
Older files often hide copied plugin rules or a forgotten Disallow: / from a staging launch. The checker shows how each AI bot reads that file today.
Finding out why you are never cited
If your brand rarely appears in ChatGPT search or Perplexity answers, confirm OAI-SearchBot and PerplexityBot can reach your pages before blaming content quality.
Opting out of training only
Publishers who want visibility in AI answers but not their content in training data can block GPTBot, ClaudeBot, CCBot and similar bots while keeping search bots open, then verify the split here.
Checking CDN bot settings
Cloudflare and other CDNs offer one-click AI bot blocking. The firewall test shows whether those rules are active and which crawlers they catch, which robots.txt alone will never tell you.
How the evaluation works, and what to block
The parser follows RFC 9309, the robots.txt standard. Each crawler obeys only the most specific user-agent group that names it, falling back to the * group if none does; rules from different groups are not combined. Within that group, the longest matching path wins, * matches any sequence of characters, $ anchors the end of the URL, and when an Allow and a Disallow rule match with equal length, Allow wins.
The HTTP status of robots.txt changes everything. A 4xx response, including 404, means crawlers treat the whole site as allowed. A 5xx response means compliant crawlers treat the site as fully disallowed until the file becomes reachable again, so a misconfigured server can quietly remove you from AI search.
Blocking is a business decision, not a technical default. Blocking training crawlers does not remove anything already collected. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those assistants' answers and citations. Google-Extended controls whether Google can use your content for Gemini models; it does not affect your Google Search rankings or your eligibility for AI Overviews, which are built from the Googlebot index.
Remember that robots.txt is voluntary. Some providers state that user-triggered fetchers may not apply robots.txt rules because a person requested the page. If you need enforcement, use firewall rules, and note that verified-bot rules matched by IP address can behave differently from the user-agent test this tool runs.
Frequently asked questions
Is the AI crawler checker free?
Yes, the AI crawler checker is free, with no account required. You can check live domains, test individual paths and paste draft robots.txt files as often as you need. It is built for site owners, SEO teams and agencies who want a quick, accurate answer on AI bot access without installing software or reading through raw server logs.
What does the tool fetch from my site, and is anything stored?
The tool fetches only public resources: your robots.txt, the page you entered and /llms.txt, plus a few requests to that page using AI crawler user agents for the firewall test. Draft robots.txt text in paste mode is evaluated for your session and is not published or shared.
Will blocking GPTBot remove my site from ChatGPT search?
No, blocking GPTBot does not remove you from ChatGPT search results. GPTBot is OpenAI's training crawler, while OAI-SearchBot handles indexing for search features, and each obeys its own robots.txt rules. To appear in ChatGPT answers with citations, keep OAI-SearchBot allowed.
Does blocking Google-Extended hurt my Google rankings?
No, blocking Google-Extended has no effect on Google Search rankings or AI Overviews. It is a control token, not a separate crawler, and it governs whether content Googlebot collects can be used for Gemini models. AI Overviews are generated from the regular Search index, so the only way to leave them is through Googlebot or snippet controls such as nosnippet.
Why does robots.txt say allowed but the firewall test says blocked?
That result means a CDN, WAF or server rule is blocking the bot before robots.txt is ever considered. Cloudflare's AI bot blocking, rate limits and bot-fight modes are common causes. Keep in mind that verified-bot rules based on IP may treat the real crawler differently from our user-agent test.
What happens if my robots.txt returns a 404 or 500 error?
A 404 or other 4xx response means crawlers treat your whole site as allowed, as if robots.txt contained no rules. A 500 or other 5xx response is more serious: under RFC 9309, compliant crawlers assume the entire site is disallowed. Fix server errors on robots.txt quickly, because they can block search and AI crawlers alike.
Do I need an llms.txt file?
No, llms.txt is optional and has no effect on crawler access. It is a proposed markdown file that points AI tools to your most useful pages, and major AI providers have not committed to reading it. The checker reports whether you have one because it is cheap to add. Access itself is still controlled by robots.txt and your firewall.
Built and reviewed by Umer Anees, SEO consultant.
Related tools
More AI Search tools →llms.txt Generator & Validator
Draft an llms.txt from your sitemap, then validate it against the spec.
AI Search
Word Counter
Count words, characters, sentences and reading time as you type.
Text
Case Converter
Switch text between UPPER, lower, Title, Sentence, camelCase and more.
Text
Meta Tag Generator
Write title, description, Open Graph and Twitter tags with a live SERP preview.
SEO