AI Crawlers List: Every AI Bot That Visits Your Website
AI companies send three kinds of bots to websites: crawlers that collect training data, crawlers that build their search index, and fetchers that open a page because a user asked. Each one has its own robots.txt name, so you can allow some and block others.
The AI crawlers that matter
| robots.txt name | Company | Kind | What it is for |
|---|---|---|---|
GPTBot | OpenAI | Training | Content that may be used to train OpenAI models |
OAI-SearchBot | OpenAI | Search | Pages shown and cited in ChatGPT search |
ChatGPT-User | OpenAI | User fetch | Opens a page when a ChatGPT user asks about it |
ClaudeBot | Anthropic | Training | Content that may be used to train Claude models |
Claude-SearchBot | Anthropic | Search | Indexes pages to improve Claude's search results |
Claude-User | Anthropic | User fetch | Opens a page when a Claude user asks a question |
PerplexityBot | Perplexity | Search | Surfaces and links sites in Perplexity answers |
Perplexity-User | Perplexity | User fetch | Visits a page a user's question needs |
Google-Extended | Control token | Whether Google may use your content for Gemini training and grounding. It does not crawl on its own and does not affect Google Search | |
Googlebot | Search | Google Search and all its features, including AI Overviews | |
Bingbot | Microsoft | Search | Bing search, which Copilot answers draw on |
Applebot | Apple | Search | Siri and Spotlight suggestions |
Applebot-Extended | Apple | Control token | Whether Apple may use pages Applebot crawled to train its AI models |
CCBot | Common Crawl | Training | A free public web archive that many AI models are trained on |
meta-externalagent | Meta | Training | Content for Meta's AI models and products |
Amazonbot | Amazon | Search | Amazon services such as Alexa answers |
Bytespider | ByteDance | Training | ByteDance's web crawler |
DuckAssistBot | DuckDuckGo | User fetch | Pages used in DuckDuckGo's AI-assisted answers |
"Training" crawlers shape what a model knows, with no link back to you. "Search" crawlers and "user fetch" agents are the ones that put your pages in answers with links, so they are the ones that send visitors.
A robots.txt that keeps you visible in AI answers
This lets every AI search engine and assistant read your site and keeps your content out of model training. Delete the second group if you are fine with training too.
# AI search and answer engines: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
Allow: /
# AI model training: blocked (optional)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /Several User-agent lines in a row share the rules below them, which keeps the file short. Leave Googlebot, Bingbot and Applebot alone unless you want to leave those search engines entirely.
Do user fetchers follow robots.txt?
Not always, and the companies say so. OpenAI says that for ChatGPT-User, "because these actions are initiated by a user, robots.txt rules may not apply", and Perplexity says Perplexity-User "generally ignores robots.txt rules". The reasoning is that a person asked for the page, the way a browser would. Training and search crawlers do follow robots.txt.
How to tell a real AI crawler from a fake one
Anyone can put "GPTBot" in a user-agent string. The big operators publish the IP addresses their bots use, so you can check:
- OpenAI: gptbot.json, searchbot.json, chatgpt-user.json
- Anthropic: claude.com/crawling/bots.json
- Perplexity: perplexitybot.json, perplexity-user.json
Blocked without knowing it
robots.txt is not the only gate. Firewalls, bot-protection services and some hosting plans refuse AI crawlers on their own, and nothing in robots.txt shows it. If an assistant never mentions your site, test the site as each bot. The free AI visibility checker requests your homepage as ten of the crawlers above and reports which ones get through, which are blocked by robots.txt, and which are stopped by a firewall.
Can AI assistants read your site?
Check your homepage as ten AI crawlers, including OAI-SearchBot, ClaudeBot and PerplexityBot. Free, about 30 seconds, no signup.