PerplexityBot: User Agents, IPs, robots.txt Rules and Log Checks

PerplexityBot is Perplexity’s web crawler: it finds and indexes pages so Perplexity can surface and link them in its answers, and Perplexity says it is not used to train AI models. You control it in robots.txt with User-agent: PerplexityBot. A second agent, Perplexity-User, fetches pages when someone asks a question, and it generally ignores robots.txt.

This guide covers both agents, their user agent strings and IP lists, copy-paste robots.txt rules, firewall setup, a script that tells real Perplexity traffic from fakes, the stealth crawling dispute, the claims where the popular guides contradict the official docs, and how to find out whether letting the bot in produces any business at all.

What Is PerplexityBot?

Perplexity is an answer engine. It replies to questions with a written answer and numbered links to the pages it used. To have pages to choose from, it keeps its own index of the web, and PerplexityBot is the crawler that builds and refreshes that index.

Perplexity’s crawler documentation describes it in three statements that matter for site owners:

• It is designed to surface and link websites in Perplexity’s search results.
• It is not used to crawl content for AI foundation models.
• If you want to appear in those results, Perplexity recommends allowing PerplexityBot in robots.txt and permitting requests from its published IP ranges.

It is a search crawler, not a training crawler. Blocking it removes you from the pool of pages Perplexity can cite, and it does not buy you any training protection, because Perplexity documents no separate training crawler to begin with.

PerplexityBot vs Perplexity-User

Perplexity documents two agents. Most robots.txt mistakes come from treating them as one.

A Perplexity-User hit is the closer signal that your page was used in a live answer. A PerplexityBot hit only means your page was indexed or refreshed.

The crawler page lists only these two agents. It says nothing about how Perplexity’s browser agent identifies itself when it browses for a user, so do not assume these two tokens cover every request Perplexity products make.

User Agent Strings and IP Ranges

The full user agent strings, as published in Perplexity’s docs:

In robots.txt you use only the product token: PerplexityBot or Perplexity-User. Matching is case-insensitive.

Perplexity publishes its IP addresses as JSON, one file per agent, at https://www.perplexity.com/perplexitybot.json and https://www.perplexity.com/perplexity-user.json. When we fetched them on October 5, 2026, the PerplexityBot file listed 8 IPv4 prefixes (created February 7, 2025) and the Perplexity-User file listed 4 (created October 17, 2025). The lists change, so read them from the URL each time instead of copying them into a config file once.

The user agent proves nothing on its own. Anyone can send PerplexityBot/1.0 in a header. A request is genuinely from PerplexityBot only when the user agent and the source IP both match.

How to Allow or Block PerplexityBot in robots.txt

Your robots.txt lives at the root of the host, for example https://example.com/robots.txt. Each subdomain needs its own file. Perplexity says each setting works independently and changes can take up to 24 hours to show.

Allow everything

If your file has no rule that blocks PerplexityBot, it is already allowed. An explicit group is useful when a User-agent: * group blocks bots by default. Read the next section before you add it.

Block everything

This keeps the declared crawler out and, over time, your pages out of Perplexity’s index. It does not stop Perplexity-User. To keep that agent out too, you need a firewall rule (covered below).

Block some paths

Under the Robots Exclusion Protocol (RFC 9309), the most specific rule wins, meaning the matching rule with the longest path. Disallow: /members/ beats Allow: / for anything under /members/, whatever order you write them in. When an Allow and a Disallow are equally long, Allow wins.

The robots.txt Mistake That Opens Your Admin Pages

Here is a common file. The site blocks private sections for every bot, then adds a group to let PerplexityBot in:

It looks harmless. It is not. RFC 9309 says a crawler obeys the group that matches its product token, and falls back to the * group only when no group matches. Once a PerplexityBot group exists, PerplexityBot ignores the * group entirely. In this file PerplexityBot may crawl /admin/, /cart/ and your internal search pages, because its own group allows everything.

The fix is to repeat every rule you still want inside the specific group:

Two more rules from the standard that save debugging time:

• Order of groups does not matter. Putting the PerplexityBot group above or below the * group changes nothing. Group selection is by user agent, not position.
• Duplicate groups merge. If two groups both name PerplexityBot, the crawler combines their rules into one group. Search the whole file before you assume a rule is the only one.

The same logic applies to every crawler you name: GPTBot, ClaudeBot, Googlebot. Any time you add a named group, copy the restrictions from * into it.

Firewall and CDN Rules for Perplexity

robots.txt is a request. A web application firewall (WAF) is enforcement. Perplexity’s docs say sites behind a WAF may need to allow its agents explicitly and give the steps:

• Cloudflare WAF: create a rule where the user agent contains PerplexityBot or Perplexity-User and the source IP is in the ranges from the two JSON files, then set the action (allow or skip).
• AWS WAF: create IP sets from the JSON files, add string match conditions for the two user agents, create allow rules and associate them with your web ACL.

To block instead of allow, use the same conditions with a block action. A rule on the user agent alone stops the declared agents, which is all robots.txt does too. A rule that also requires the published IPs to allow traffic means a fake PerplexityBot from another network gets no special treatment.

Crawl rate. Perplexity’s docs do not mention the Crawl-delay directive, and RFC 9309 does not define it, so do not count on it. If Perplexity traffic loads your server, rate limit at the CDN or server and return 429 with a Retry-After header.

Cloudflare specifically. In its August 2025 report, Cloudflare said it had de-listed Perplexity as a verified bot. If you rely on a “verified bots” allow rule, Perplexity may not be in it. Look at the security events for requests with Perplexity user agents and see what action was applied.

Verify Real PerplexityBot Traffic in Your Logs

Google Search Console only reports Google’s own crawlers, so it shows nothing about Perplexity. Your web server access logs, or your CDN’s logs and bot dashboard, are the record. A quick look, for the common combined log format:

That prints the most frequent IP, status code and path combinations for both agents. It still trusts the user agent. To separate real Perplexity traffic from impostors, check every IP against the published CIDR ranges. This Python 3 script uses only the standard library, downloads both lists, and counts hits by agent, verdict and status code:

Run it with python3 check_perplexity.py /var/log/nginx/access.log. On a four-line test log it printed:

How to read the result:

• Verified with 200: Perplexity reaches your pages. Check that a /robots.txt request appears before the crawl, which shows it read your rules.
• Verified with 403, 429 or 503: your firewall, CDN or rate limiter is turning Perplexity away, whatever robots.txt says.
• Spoofed: something is using the PerplexityBot name from an IP outside the list. Treat it like any unknown scraper.
• No hits at all: either you are blocked upstream (CDN rules apply before your server logs) or Perplexity has not crawled you. Check the CDN’s logs before concluding anything.

If your log format differs, adjust the regular expression. The IP check is the part that matters.

When PerplexityBot Is Allowed but Still Cannot Crawl

robots.txt itself can fail in ways that change what a crawler is allowed to do. RFC 9309 sets the rules, and they surprise people:

The Stealth Crawling Dispute

Two public reports say Perplexity reached sites that had blocked it, using requests that did not identify as Perplexity.

June 2024. Developer Robb Knight, then Wired, reported Perplexity fetching pages on sites that disallowed PerplexityBot, from cloud IPs with a generic browser user agent. As reported at the time, Perplexity said it did not ignore robots.txt and attributed the traffic to a third-party crawling provider.

August 2025. Cloudflare’s investigation set up brand-new domains that disallowed all crawling in robots.txt and blocked both Perplexity agents with WAF rules. It then asked Perplexity about them and got detailed answers. Cloudflare reported:

• The declared Perplexity-User agent made 20 to 25 million requests a day across its network.
• An undeclared crawler, posing as Chrome on macOS, made 3 to 6 million a day, from IPs outside Perplexity’s published ranges and rotating across networks (ASNs).
• Cloudflare de-listed Perplexity as a verified bot and added signatures for the undeclared crawler to its managed rule that blocks AI crawling, which is available to free plans.
• When the stealth crawler was blocked too, Perplexity’s answers about the test sites became vaguer and lacked details from the original pages.

The practical lesson does not depend on who is right. robots.txt only binds requests that identify themselves. If content must not be read by any automated system, put it behind a login or a bot management product that fingerprints traffic, not just a robots.txt line.

Should You Allow or Block PerplexityBot?

PerplexityBot does not affect Google. Googlebot reads only its own groups and the * group, and Perplexity does not pass any ranking signal to Google. So the decision is only about Perplexity.

If you allow it to get cited, getting crawled is only the entry ticket. What earns citations is covered in our Perplexity SEO guide.

How to Stop AI Crawlers (and What robots.txt Cannot Do)

Most AI companies run separate agents for training, for search indexing and for live user requests. Blocking one does not block the others. The main tokens, all used in robots.txt as User-agent: <token>:

Vendors rename and add agents, so check each company’s crawler page before you rely on a list, including this one. Then accept the limits: robots.txt is honored only by crawlers that choose to, user-triggered fetchers are often exempt by design, and RFC 9309 states plainly that its rules are not a form of access authorization. For anything that must stay private, use authentication. For everything else, a firewall rule on user agent plus IP backs up what robots.txt asks.

Where the PerplexityBot Guides Contradict Each Other

We read the ten pages ranking for “perplexitybot” and compared them with Perplexity’s documentation and RFC 9309. Several popular claims are wrong or out of date:

One question we could not settle from a primary source: how Perplexity responded to the 2025 report. One ranking guide says Perplexity acknowledged the issue and committed to stricter compliance; another says it disputed the findings. Neither links a Perplexity statement, so treat both as unconfirmed.

Is Allowing PerplexityBot Worth It? Measure the Outcome

Crawler hits are not visitors. A thousand PerplexityBot requests can produce zero customers. To decide whether access pays off, connect three numbers per landing page:

1. Perplexity-User fetches from your verified logs: a rough proxy for how often a page is pulled into live answers.
2. Sessions referred by perplexity.ai, from your analytics landing page report. Our guide to AI traffic analytics shows how to isolate them in GA4. Visits from apps that send no referrer will land in Direct, so this count is a floor.
3. Conversions and their value on those sessions: demo requests, purchases, calls, each with a dollar value.

This only works if you track conversions per landing page and per AI source, not just traffic. Our AI conversion tracking guide walks through that setup. SEOConversion does it with one cookieless script: it identifies Perplexity visits from the referrer, records the conversions on each landing page and turns them into value, and leaves visits that arrive without a referrer in Direct instead of guessing.

FAQ