About Am I Crawlable

AI assistants and AI search engines now decide whether your site gets read, cited, and recommended — and whether they can even reach it is surprisingly hard to know. Blocks hide in two layers: the rules in your robots.txt, and server or CDN-level bot protection (such as Cloudflare's default AI-crawler blocking) that a robots.txt checker never sees. Am I Crawlable tests both.

How the check works

When you check a URL, our server fetches the site's robots.txt and parses it against the verified user-agent tokens of the AI and search crawlers that matter (each documented in our crawler directory with its operator's primary source and a verification date). It then probes the homepage with each crawler's user-agent string and compares the responses to a normal browser request — surfacing server-level blocks that robots.txt analysis cannot detect.

Honest results, including their limits

Our probes are unverified requests carrying a crawler's user-agent — they come from our server, not from OpenAI's or Anthropic's real IP ranges. Some bot-protection systems treat unverified look-alikes differently from the real, verified crawler. The report says exactly what was observed and flags cases where the real crawler's treatment may differ, rather than pretending certainty.

What we don't do

Am I Crawlable is an independent technical diagnostic and is not affiliated with OpenAI, Anthropic, Perplexity, Google, Microsoft, or Cloudflare. It reports the technical reachability of a site; it does not provide legal advice about content licensing or crawling policy. We do not crawl or index anyone's site: checks run only when you request them, touch only robots.txt and the page you name, and the URLs you check are not stored in analytics.