TavilyBot
Tavily •
Since 2024
Quick Actions
Official Docs
What is TavilyBot?
TavilyBot is the web fetcher operated by Tavily, a company that provides real-time web data APIs for AI agents and applications. When AI agents or applications use Tavily's API to research topics, answer questions, or complete tasks, TavilyBot fetches the relevant web content. It serves as the infrastructure layer between AI systems and the web, enabling AI agents to access current information. The bot respects robots.txt directives.
User Agent String
TavilyBot
How to Control TavilyBot
Block Completely
To prevent TavilyBot from accessing your entire website, add this to your robots.txt file:
# Block TavilyBot
User-agent: TavilyBot
Disallow: /
Block Specific Directories
To restrict access to certain parts of your site while allowing others:
User-agent: TavilyBot
Disallow: /admin/
Disallow: /private/
Disallow: /wp-admin/
Allow: /public/
Set Crawl Delay
To slow down the crawl rate (note: not all bots respect this directive):
User-agent: TavilyBot
Crawl-delay: 10
How to Verify TavilyBot
Verification Method:
Verify requests originate from Tavily IP ranges
Verify requests originate from Tavily IP ranges
Learn more in the official documentation.
⚠️ AI Training Notice
This bot may collect and use your website content for AI model training. Consider whether you want your content used for this purpose before allowing access.
This bot may collect and use your website content for AI model training. Consider whether you want your content used for this purpose before allowing access.
Detection Patterns
Multiple ways to detect TavilyBot in your application:
Basic Pattern
/TavilyBot/iStrict Pattern
/^TavilyBot$/Flexible Pattern
/TavilyBot[\s\/]?[\d\.]*?/iVendor Match
/.*Tavily.*TavilyBot/iImplementation Examples
// PHP Detection for TavilyBot
function detect_tavilybot() {
$user_agent = $_SERVER['HTTP_USER_AGENT'] ?? '';
$pattern = '/TavilyBot/i';
if (preg_match($pattern, $user_agent)) {
// Log the detection
error_log('TavilyBot detected from IP: ' . $_SERVER['REMOTE_ADDR']);
// Set cache headers
header('Cache-Control: public, max-age=3600');
header('X-Robots-Tag: noarchive');
// Optional: Serve cached version
if (file_exists('cache/' . md5($_SERVER['REQUEST_URI']) . '.html')) {
readfile('cache/' . md5($_SERVER['REQUEST_URI']) . '.html');
exit;
}
return true;
}
return false;
}
# Python/Flask Detection for TavilyBot
import re
from flask import request, make_responsedef detect_tavilybot():
user_agent = request.headers.get('User-Agent', '')
pattern = r'TavilyBot'
if re.search(pattern, user_agent, re.IGNORECASE):
# Create response with caching
response = make_response()
response.headers['Cache-Control'] = 'public, max-age=3600'
response.headers['X-Robots-Tag'] = 'noarchive'
return True
return False# Django Middleware
class TavilyBotMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
if self.detect_bot(request):
# Handle bot traffic
pass
return self.get_response(request)
// JavaScript/Node.js Detection for TavilyBot
const express = require('express');
const app = express();// Middleware to detect TavilyBot
function detectTavilyBot(req, res, next) {
const userAgent = req.headers['user-agent'] || '';
const pattern = /TavilyBot/i;
if (pattern.test(userAgent)) {
// Log bot detection
console.log('TavilyBot detected from IP:', req.ip);
// Set cache headers
res.set({
'Cache-Control': 'public, max-age=3600',
'X-Robots-Tag': 'noarchive'
});
// Mark request as bot
req.isBot = true;
req.botName = 'TavilyBot';
}
next();
}app.use(detectTavilyBot);
# Apache .htaccess rules for TavilyBot# Block completely
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} TavilyBot [NC]
RewriteRule .* - [F,L]# Or redirect to a static version
RewriteCond %{HTTP_USER_AGENT} TavilyBot [NC]
RewriteCond %{REQUEST_URI} !^/static/
RewriteRule ^(.*)$ /static/$1 [L]# Or set environment variable for PHP
SetEnvIfNoCase User-Agent "TavilyBot" is_bot=1# Add cache headers for this bot
<If "%{HTTP_USER_AGENT} =~ /TavilyBot/i">
Header set Cache-Control "public, max-age=3600"
Header set X-Robots-Tag "noarchive"
</If>
# Nginx configuration for TavilyBot# Map user agent to variable
map $http_user_agent $is_tavilybot {
default 0;
~*TavilyBot 1;
}server {
# Block the bot completely
if ($is_tavilybot) {
return 403;
}
# Or serve cached content
location / {
if ($is_tavilybot) {
root /var/www/cached;
try_files $uri $uri.html $uri/index.html @backend;
}
try_files $uri @backend;
}
# Add headers for bot requests
location @backend {
if ($is_tavilybot) {
add_header Cache-Control "public, max-age=3600";
add_header X-Robots-Tag "noarchive";
}
proxy_pass http://backend;
}
}
Should You Block This Bot?
Recommendations based on your website type:
| Site Type | Recommendation | Reasoning |
|---|---|---|
| E-commerce | Limit Access | Protect pricing and inventory data from AI training |
| Blog/News | Consider Blocking | Your content may be used for AI training without compensation |
| SaaS Application | Block | No benefit for application interfaces; preserve resources |
| Documentation | Selective | Allow for public docs, block for internal docs |
| Corporate Site | Limit | Allow for public pages, block sensitive areas like intranets |
Advanced robots.txt Configurations
E-commerce Site Configuration
User-agent: TavilyBot
Crawl-delay: 5
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /api/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&page=
Allow: /products/
Allow: /categories/
Sitemap: https://example.com/sitemap.xml
Publishing/Blog Configuration
User-agent: TavilyBot
# Blocking AI training bot
Disallow: /
SaaS/Application Configuration
User-agent: TavilyBot
Disallow: /app/
Disallow: /api/
Disallow: /dashboard/
Disallow: /settings/
Allow: /
Allow: /pricing/
Allow: /features/
Allow: /docs/
Quick Reference
User Agent Match
TavilyBotRobots.txt Name
TavilyBotCategory
aiRespects robots.txt
Yes
Copied to clipboard!