AI assistants increasingly decide whether your site gets read and cited. Whether they can reach you is governed by two independent layers, and most "AI crawler checkers" only look at one.
Layer 1: robots.txt
Your robots.txt names which crawlers may fetch which paths. Each AI operator publishes distinct tokens — OpenAI's GPTBot/OAI-SearchBot/ChatGPT-User, Anthropic's ClaudeBot/Claude-SearchBot/Claude-User, Perplexity's PerplexityBot/Perplexity-User (all verified in our crawler directory). A common mistake is blocking a training bot (GPTBot) while assuming you've blocked AI search — you haven't; those are separate tokens.
Layer 2: your server / CDN
Even with a permissive robots.txt, your host or CDN can block bots at the network edge. Since Cloudflare's September 15, 2026 default blocking, this is common — and invisible to robots.txt tools.
Check both in one pass
Our live crawler check reads your robots.txt AND probes your homepage with each crawler's user-agent, comparing the response to a normal browser — surfacing edge-level blocks. One honest caveat: our probes carry a crawler's user-agent but come from our server, not the crawler's real IP ranges, so a bot-protection system that verifies IPs may treat the real crawler differently; the report says so where it matters.