PerplexityBot: Perplexity's Search Crawler, User Agent and robots.txt
PerplexityBot is the crawler Perplexity uses to surface and link websites in its search results. Perplexity says it is not used to crawl content for AI foundation models, so for most sites it is a bot worth letting in.
PerplexityBotWhat PerplexityBot does
Perplexity says PerplexityBot surfaces and links websites in Perplexity's search results, and that it is "not used to crawl content for AI foundation models." Perplexity answers almost every question with numbered sources, so the pages this bot can read are the pages Perplexity can cite and send visitors to.
Perplexity runs a second agent, Perplexity-User, for a different job:
| Agent | What it does | robots.txt |
|---|---|---|
| PerplexityBot | Crawls to surface and link sites in Perplexity search | Controlled with the PerplexityBot token |
| Perplexity-User | Visits a page when a user's question needs it, to answer and link to it | "Generally ignores robots.txt rules", since a user asked for the page |
PerplexityBot user agent
The full user-agent string, as Perplexity documents it:
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)And for Perplexity-User:
Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)To confirm a request is genuine, check its IP address against perplexity.ai/perplexitybot.json (and perplexity-user.json for the user agent).
How to allow PerplexityBot
User-agent: PerplexityBot
Allow: /Perplexity says each setting works independently and "it may take up to 24 hours for our systems to reflect changes." If PerplexityBot still never shows up in your logs, the block is probably outside robots.txt: a firewall rule, a bot-protection service or a hosting setting. The AI visibility checker requests your homepage as PerplexityBot and Perplexity-User and shows which ones get through.
How to block PerplexityBot
User-agent: PerplexityBot
Disallow: /That removes your pages from what Perplexity's crawler collects for search. Keep in mind that Perplexity-User may still open a page a user specifically asks about, because Perplexity treats those as user requests.
Why it matters
Perplexity's answers lead with citations, so being readable by PerplexityBot is close to a requirement for being recommended there. Allowing it costs nothing for most sites and, unlike training crawlers, comes with links back. For the other assistants, see the AI crawlers list.
Source: Perplexity, "Perplexity Crawlers".
Can AI assistants read your site?
Check your homepage as ten AI crawlers, including OAI-SearchBot, ClaudeBot and PerplexityBot. Free, about 30 seconds, no signup.