heritrix
Internet Archive •
Since 2004
Quick Actions
Official Docs
What is heritrix?
Heritrix is an open-source, extensible web crawler designed for web archiving. Developed by the Internet Archive, it is used by libraries, research institutions, and archives worldwide to preserve web content. Heritrix is highly configurable and can handle large-scale crawls with millions of URLs. It respects robots.txt directives. While primarily associated with the Internet Archive, many organizations run their own Heritrix instances for archival purposes.
User Agent String
Mozilla/5.0 (compatible; heritrix/3.4.0 +http://archive.org)
How to Control heritrix
Block Completely
To prevent heritrix from accessing your entire website, add this to your robots.txt file:
# Block heritrix
User-agent: heritrix
Disallow: /
Block Specific Directories
To restrict access to certain parts of your site while allowing others:
User-agent: heritrix
Disallow: /admin/
Disallow: /private/
Disallow: /wp-admin/
Allow: /public/
Set Crawl Delay
To slow down the crawl rate (note: not all bots respect this directive):
User-agent: heritrix
Crawl-delay: 10
How to Verify heritrix
Verification Method:
Check user agent string for heritrix identifier
Check user agent string for heritrix identifier
Learn more in the official documentation.
Detection Patterns
Multiple ways to detect heritrix in your application:
Basic Pattern
/heritrix/iStrict Pattern
/^Mozilla/5\.0 \(compatible; heritrix/3\.4\.0 \+http\://archive\.org\)$/Flexible Pattern
/heritrix[\s\/]?[\d\.]*?/iVendor Match
/.*Internet Archive.*heritrix/iImplementation Examples
// PHP Detection for heritrix
function detect_heritrix() {
$user_agent = $_SERVER['HTTP_USER_AGENT'] ?? '';
$pattern = '/heritrix/i';
if (preg_match($pattern, $user_agent)) {
// Log the detection
error_log('heritrix detected from IP: ' . $_SERVER['REMOTE_ADDR']);
// Set cache headers
header('Cache-Control: public, max-age=3600');
header('X-Robots-Tag: noarchive');
// Optional: Serve cached version
if (file_exists('cache/' . md5($_SERVER['REQUEST_URI']) . '.html')) {
readfile('cache/' . md5($_SERVER['REQUEST_URI']) . '.html');
exit;
}
return true;
}
return false;
}
# Python/Flask Detection for heritrix
import re
from flask import request, make_responsedef detect_heritrix():
user_agent = request.headers.get('User-Agent', '')
pattern = r'heritrix'
if re.search(pattern, user_agent, re.IGNORECASE):
# Create response with caching
response = make_response()
response.headers['Cache-Control'] = 'public, max-age=3600'
response.headers['X-Robots-Tag'] = 'noarchive'
return True
return False# Django Middleware
class heritrixMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
if self.detect_bot(request):
# Handle bot traffic
pass
return self.get_response(request)
// JavaScript/Node.js Detection for heritrix
const express = require('express');
const app = express();// Middleware to detect heritrix
function detectheritrix(req, res, next) {
const userAgent = req.headers['user-agent'] || '';
const pattern = /heritrix/i;
if (pattern.test(userAgent)) {
// Log bot detection
console.log('heritrix detected from IP:', req.ip);
// Set cache headers
res.set({
'Cache-Control': 'public, max-age=3600',
'X-Robots-Tag': 'noarchive'
});
// Mark request as bot
req.isBot = true;
req.botName = 'heritrix';
}
next();
}app.use(detectheritrix);
# Apache .htaccess rules for heritrix# Block completely
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} heritrix [NC]
RewriteRule .* - [F,L]# Or redirect to a static version
RewriteCond %{HTTP_USER_AGENT} heritrix [NC]
RewriteCond %{REQUEST_URI} !^/static/
RewriteRule ^(.*)$ /static/$1 [L]# Or set environment variable for PHP
SetEnvIfNoCase User-Agent "heritrix" is_bot=1# Add cache headers for this bot
<If "%{HTTP_USER_AGENT} =~ /heritrix/i">
Header set Cache-Control "public, max-age=3600"
Header set X-Robots-Tag "noarchive"
</If>
# Nginx configuration for heritrix# Map user agent to variable
map $http_user_agent $is_heritrix {
default 0;
~*heritrix 1;
}server {
# Block the bot completely
if ($is_heritrix) {
return 403;
}
# Or serve cached content
location / {
if ($is_heritrix) {
root /var/www/cached;
try_files $uri $uri.html $uri/index.html @backend;
}
try_files $uri @backend;
}
# Add headers for bot requests
location @backend {
if ($is_heritrix) {
add_header Cache-Control "public, max-age=3600";
add_header X-Robots-Tag "noarchive";
}
proxy_pass http://backend;
}
}
Should You Block This Bot?
Recommendations based on your website type:
| Site Type | Recommendation | Reasoning |
|---|---|---|
| E-commerce | Optional | Evaluate based on bandwidth usage vs. benefits |
| Blog/News | Allow | Increases content reach and discoverability |
| SaaS Application | Block | No benefit for application interfaces; preserve resources |
| Documentation | Selective | Allow for public docs, block for internal docs |
| Corporate Site | Limit | Allow for public pages, block sensitive areas like intranets |
Advanced robots.txt Configurations
E-commerce Site Configuration
User-agent: heritrix
Crawl-delay: 5
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /api/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&page=
Allow: /products/
Allow: /categories/
Sitemap: https://example.com/sitemap.xml
Publishing/Blog Configuration
User-agent: heritrix
Crawl-delay: 10
Disallow: /wp-admin/
Disallow: /drafts/
Disallow: /preview/
Disallow: /*?replytocom=
Allow: /
SaaS/Application Configuration
User-agent: heritrix
Disallow: /app/
Disallow: /api/
Disallow: /dashboard/
Disallow: /settings/
Allow: /
Allow: /pricing/
Allow: /features/
Allow: /docs/
Quick Reference
User Agent Match
heritrixRobots.txt Name
heritrixCategory
otherRespects robots.txt
Yes
Copied to clipboard!