Cookie Consent by Free Privacy Policy Generator PerplexityBot User Agent - Perplexity AI Bot Details | CL SEO

PerplexityBot

Perplexity AI Since 2022
Ai Search Respects robots.txt
#ai #search #answer-engine #crawler
Quick Actions
Official Docs

What is PerplexityBot?

PerplexityBot is the web crawler used by Perplexity AI, an AI-powered search and answer engine that provides direct answers to user queries with cited sources. The bot crawls web content to maintain an up-to-date index that powers Perplexity's conversational search experience. Unlike traditional search engines that return lists of links, Perplexity uses the content gathered by this bot to generate comprehensive answers with proper attribution. The crawler respects robots.txt directives and follows responsible crawling practices. For publishers, PerplexityBot represents an opportunity to have their content featured in AI-generated answers with proper citation, potentially driving traffic through source attribution.

User Agent String

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/bot)

How to Control PerplexityBot

Block Completely

To prevent PerplexityBot from accessing your entire website, add this to your robots.txt file:

# Block PerplexityBot User-agent: PerplexityBot Disallow: /

Block Specific Directories

To restrict access to certain parts of your site while allowing others:

User-agent: PerplexityBot Disallow: /admin/ Disallow: /private/ Disallow: /wp-admin/ Allow: /public/

Set Crawl Delay

To slow down the crawl rate (note: not all bots respect this directive):

User-agent: PerplexityBot Crawl-delay: 10

How to Verify PerplexityBot

Verification Method:
Check for Perplexity AI IP ranges and user agent

Learn more in the official documentation.

⚠️ AI Training Notice
This bot may collect and use your website content for AI model training. Consider whether you want your content used for this purpose before allowing access.

Detection Patterns

Multiple ways to detect PerplexityBot in your application:

Basic Pattern

/PerplexityBot/i

Strict Pattern

/^Mozilla/5\.0 AppleWebKit/537\.36 \(KHTML, like Gecko; compatible; PerplexityBot/1\.0; \+https\://perplexity\.ai/bot\)$/

Flexible Pattern

/PerplexityBot[\s\/]?[\d\.]*?/i

Vendor Match

/.*Perplexity AI.*PerplexityBot/i

Implementation Examples

// PHP Detection for PerplexityBot function detect_perplexitybot() { $user_agent = $_SERVER['HTTP_USER_AGENT'] ?? ''; $pattern = '/PerplexityBot/i'; if (preg_match($pattern, $user_agent)) { // Log the detection error_log('PerplexityBot detected from IP: ' . $_SERVER['REMOTE_ADDR']); // Set cache headers header('Cache-Control: public, max-age=3600'); header('X-Robots-Tag: noarchive'); // Optional: Serve cached version if (file_exists('cache/' . md5($_SERVER['REQUEST_URI']) . '.html')) { readfile('cache/' . md5($_SERVER['REQUEST_URI']) . '.html'); exit; } return true; } return false; }
# Python/Flask Detection for PerplexityBot import re from flask import request, make_responsedef detect_perplexitybot(): user_agent = request.headers.get('User-Agent', '') pattern = r'PerplexityBot' if re.search(pattern, user_agent, re.IGNORECASE): # Create response with caching response = make_response() response.headers['Cache-Control'] = 'public, max-age=3600' response.headers['X-Robots-Tag'] = 'noarchive' return True return False# Django Middleware class PerplexityBotMiddleware: def __init__(self, get_response): self.get_response = get_response def __call__(self, request): if self.detect_bot(request): # Handle bot traffic pass return self.get_response(request)
// JavaScript/Node.js Detection for PerplexityBot const express = require('express'); const app = express();// Middleware to detect PerplexityBot function detectPerplexityBot(req, res, next) { const userAgent = req.headers['user-agent'] || ''; const pattern = /PerplexityBot/i; if (pattern.test(userAgent)) { // Log bot detection console.log('PerplexityBot detected from IP:', req.ip); // Set cache headers res.set({ 'Cache-Control': 'public, max-age=3600', 'X-Robots-Tag': 'noarchive' }); // Mark request as bot req.isBot = true; req.botName = 'PerplexityBot'; } next(); }app.use(detectPerplexityBot);
# Apache .htaccess rules for PerplexityBot# Block completely RewriteEngine On RewriteCond %{HTTP_USER_AGENT} PerplexityBot [NC] RewriteRule .* - [F,L]# Or redirect to a static version RewriteCond %{HTTP_USER_AGENT} PerplexityBot [NC] RewriteCond %{REQUEST_URI} !^/static/ RewriteRule ^(.*)$ /static/$1 [L]# Or set environment variable for PHP SetEnvIfNoCase User-Agent "PerplexityBot" is_bot=1# Add cache headers for this bot <If "%{HTTP_USER_AGENT} =~ /PerplexityBot/i"> Header set Cache-Control "public, max-age=3600" Header set X-Robots-Tag "noarchive" </If>
# Nginx configuration for PerplexityBot# Map user agent to variable map $http_user_agent $is_perplexitybot { default 0; ~*PerplexityBot 1; }server { # Block the bot completely if ($is_perplexitybot) { return 403; } # Or serve cached content location / { if ($is_perplexitybot) { root /var/www/cached; try_files $uri $uri.html $uri/index.html @backend; } try_files $uri @backend; } # Add headers for bot requests location @backend { if ($is_perplexitybot) { add_header Cache-Control "public, max-age=3600"; add_header X-Robots-Tag "noarchive"; } proxy_pass http://backend; } }

Should You Block This Bot?

Recommendations based on your website type:

Site TypeRecommendationReasoning
E-commerce Allow Essential for product visibility in search results
Blog/News Consider Blocking Your content may be used for AI training without compensation
SaaS Application Block No benefit for application interfaces; preserve resources
Documentation Allow Improves documentation discoverability for developers
Corporate Site Allow Allow for public pages, block sensitive areas like intranets

Advanced robots.txt Configurations

E-commerce Site Configuration

User-agent: PerplexityBot Crawl-delay: 5 Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /api/ Disallow: /*?sort= Disallow: /*?filter= Disallow: /*&page= Allow: /products/ Allow: /categories/ Sitemap: https://example.com/sitemap.xml

Publishing/Blog Configuration

User-agent: PerplexityBot # Blocking AI training bot Disallow: /

SaaS/Application Configuration

User-agent: PerplexityBot Disallow: /app/ Disallow: /api/ Disallow: /dashboard/ Disallow: /settings/ Allow: / Allow: /pricing/ Allow: /features/ Allow: /docs/

Quick Reference

User Agent Match

PerplexityBot

Robots.txt Name

PerplexityBot

Category

ai, search

Respects robots.txt

Yes
Copied to clipboard!