GPTBot: OpenAI's Training Crawler, User Agent and robots.txt
GPTBot is the crawler OpenAI uses to collect content that may be used to train its generative AI models. Blocking it keeps your content out of training, but it does not remove your site from ChatGPT search, which uses a different bot.
GPTBotWhat GPTBot does
OpenAI says GPTBot crawls content that may be used to train its generative AI foundation models, and that "disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." It is a training crawler only.
OpenAI runs three bots that matter to most websites. Each one is a separate switch:
| Bot | Used for | Block it and… |
|---|---|---|
| GPTBot | Model training | Your content is not used for training |
| OAI-SearchBot | ChatGPT search results | Your pages cannot appear in ChatGPT search answers |
| ChatGPT-User | Pages a user asks ChatGPT to open | Rules "may not apply", since a person asked for the page |
GPTBot user agent
The full user-agent string, as OpenAI documents it (the version number may change):
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotFor robots.txt, use the token GPTBot. To check that a request claiming to be GPTBot is real, compare its IP address with OpenAI's published list at openai.com/gptbot.json.
How to block GPTBot
User-agent: GPTBot
Disallow: /To block only part of the site, list the folders instead:
User-agent: GPTBot
Disallow: /members/
Disallow: /drafts/OpenAI says it can take about 24 hours for its systems to pick up a robots.txt change.
Block GPTBot, stay in ChatGPT search
This is the setup most businesses want if they object to training but still want ChatGPT to recommend them. OpenAI's own example allows OAI-SearchBot while disallowing GPTBot:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /OpenAI also notes that if a site allows both bots, it "may use the results from just one crawl for both use cases", so you will not be crawled twice.
Check whether GPTBot can reach your site
A robots.txt rule is not the only thing that can block GPTBot. Firewalls, bot-protection services and some hosting plans refuse AI crawlers on their own. The free AI visibility checker requests your homepage as GPTBot, OAI-SearchBot, ChatGPT-User and seven other AI crawlers and shows exactly which ones are blocked, and by what.
Can AI assistants read your site?
Check your homepage as ten AI crawlers, including OAI-SearchBot, ClaudeBot and PerplexityBot. Free, about 30 seconds, no signup.