ClaudeBot: Anthropic's Crawler, User Agent and robots.txt Rules
ClaudeBot is the web crawler Anthropic uses to collect public pages that may be used to train its Claude models. Blocking it keeps your future content out of training. It does not control whether Claude can find your site when people search with it: that is Claude-SearchBot.
ClaudeBotWhat ClaudeBot does
Anthropic describes ClaudeBot as the crawler that "helps enhance the utility and safety of our generative AI models". It downloads public web pages that may be used to train future Claude models. When your robots.txt blocks it, Anthropic treats that as a signal to exclude your site's future material from its training datasets.
Anthropic runs three separate bots, each with its own robots.txt switch:
| Bot | What it does | If you block it |
|---|---|---|
| ClaudeBot | Collects pages that may be used for training | Your future content is excluded from training |
| Claude-SearchBot | Indexes pages to improve Claude's search results | Anthropic says it "may reduce your site's visibility and accuracy in user search results" |
| Claude-User | Opens pages when a Claude user asks a question | Claude cannot read your page for that user, which "may reduce your site's visibility for user-directed web search" |
Not to be confused with "Clawdbot", an unrelated AI assistant project that shares a similar name in search results.
ClaudeBot user agent
Anthropic's help article lists the bot by name but does not print the full user-agent string. In server logs, ClaudeBot requests identify themselves with a string ending in:
compatible; ClaudeBot/1.0; +claudebot@anthropic.comFor robots.txt you only need the token ClaudeBot. Any program can copy a user-agent string, so to confirm a request really came from Anthropic, check its IP address against the list Anthropic publishes at claude.com/crawling/bots.json.
How to block ClaudeBot
Add this to the robots.txt file at the root of your site:
User-agent: ClaudeBot
Disallow: /robots.txt works per host, so repeat it on every subdomain you want to opt out (for example blog.example.com has its own file). Anthropic also warns that blocking its IP addresses instead "may not work correctly or persistently guarantee an opt-out", so use robots.txt.
If you want Claude to keep finding your site in search while staying out of training, block ClaudeBot only and leave Claude-SearchBot and Claude-User allowed:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /How to slow ClaudeBot down
Anthropic supports the non-standard Crawl-delay rule, so you can keep ClaudeBot but limit how often it requests pages:
User-agent: ClaudeBot
Crawl-delay: 1How to allow ClaudeBot
ClaudeBot is allowed unless something blocks it. Look for three things: a Disallow rule under User-agent: ClaudeBot, a site-wide Disallow: / under User-agent: *, and firewall or CDN settings that refuse AI crawlers. The last one is invisible in robots.txt, which is why it is worth testing the site as each bot. The AI visibility checker requests your homepage as ClaudeBot, Claude-SearchBot and eight other AI crawlers and shows which ones get through.
Should you block ClaudeBot?
It depends on what you sell. Publishers whose content is the product often block training crawlers. Most businesses that want to be recommended by AI assistants allow them, because a model that has read your pages is more likely to know your brand, and because the search and user bots are what send visitors. Whatever you choose, decide it per bot: blocking ClaudeBot is a training decision, not a visibility one.
Can AI assistants read your site?
Check your homepage as ten AI crawlers, including OAI-SearchBot, ClaudeBot and PerplexityBot. Free, about 30 seconds, no signup.