AI crawler directory

The verified user-agent tokens, purposes, and official documentation of the AI and search crawlers that matter — each entry cites its operator's primary documentation and shows the date we last verified it. Crawlers change: always confirm against the official page.

• Google-Extended vs Googlebot: AI training control without losing Search

  How Google-Extended controls Gemini training/grounding separately from Google Search crawling.

  Last verified 2026-08-30

• Perplexity crawlers: PerplexityBot and Perplexity-User

  Perplexity's two crawlers — and the important catch that Perplexity-User generally ignores robots.txt.

  Last verified 2026-08-30

• Anthropic crawlers: ClaudeBot, Claude-User, Claude-SearchBot

  Anthropic's three crawlers and what blocking each does — training vs user-fetch vs search.

  Last verified 2026-08-30

• OpenAI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User

  OpenAI's three documented crawlers, their robots.txt tokens, and what blocking each one does.

  Last verified 2026-08-30