Anthropic operates three crawlers with distinct jobs, each set independently in robots.txt. Per Anthropic's documentation, all three honor standard robots.txt directives, and IP-based blocking may not reliably work — verify origin against Anthropic's published IP list instead.
What each block means
- ClaudeBot — blocking signals that your future materials should be excluded from Anthropic's AI model training datasets.
- Claude-SearchBot — blocking prevents your content being indexed for Claude's search features.
- Claude-User — blocking prevents Claude retrieving your content in response to a user's question.
To throttle rather than block, Anthropic documents the Crawl-delay directive. Confirm current tokens on Anthropic's official page before relying on them.