Verisign Spider
Verisign •
Since 2010
Quick Actions
Official Docs
What is Verisign Spider?
Verisign Spider is operated by Verisign, the global registry operator for .com and .net domains. The crawler helps Verisign monitor domain usage, identify security threats, and ensure the stability of critical internet infrastructure. As the company responsible for key parts of the DNS system, Verisign uses this crawler to detect malicious domains, phishing sites, and other security concerns. The spider plays a role in maintaining the security and reliability of the domain name system.
User Agent String
Mozilla/5.0 (compatible; Verisign Spider; +http://www.verisign.com/)
How to Control Verisign Spider
Block Completely
To prevent Verisign Spider from accessing your entire website, add this to your robots.txt file:
# Block Verisign Spider
User-agent: Verisign Spider
Disallow: /
Block Specific Directories
To restrict access to certain parts of your site while allowing others:
User-agent: Verisign Spider
Disallow: /admin/
Disallow: /private/
Disallow: /wp-admin/
Allow: /public/
Set Crawl Delay
To slow down the crawl rate (note: not all bots respect this directive):
User-agent: Verisign Spider
Crawl-delay: 10
How to Verify Verisign Spider
Verification Method:
Verisign IP ranges
Verisign IP ranges
Learn more in the official documentation.
Detection Patterns
Multiple ways to detect Verisign Spider in your application:
Basic Pattern
/Verisign Spider/i
Strict Pattern
/^Mozilla/5\.0 \(compatible; Verisign Spider; \+http\://www\.verisign\.com/\)$/
Flexible Pattern
/Verisign Spider[\s\/]?[\d\.]*?/i
Vendor Match
/.*Verisign.*Verisign/i
Implementation Examples
// PHP Detection for Verisign Spider
function detect_verisign_spider() {
$user_agent = $_SERVER['HTTP_USER_AGENT'] ?? '';
$pattern = '/Verisign Spider/i';
if (preg_match($pattern, $user_agent)) {
// Log the detection
error_log('Verisign Spider detected from IP: ' . $_SERVER['REMOTE_ADDR']);
// Set cache headers
header('Cache-Control: public, max-age=3600');
header('X-Robots-Tag: noarchive');
// Optional: Serve cached version
if (file_exists('cache/' . md5($_SERVER['REQUEST_URI']) . '.html')) {
readfile('cache/' . md5($_SERVER['REQUEST_URI']) . '.html');
exit;
}
return true;
}
return false;
}
# Python/Flask Detection for Verisign Spider
import re
from flask import request, make_response
def detect_verisign_spider():
user_agent = request.headers.get('User-Agent', '')
pattern = r'Verisign Spider'
if re.search(pattern, user_agent, re.IGNORECASE):
# Create response with caching
response = make_response()
response.headers['Cache-Control'] = 'public, max-age=3600'
response.headers['X-Robots-Tag'] = 'noarchive'
return True
return False
# Django Middleware
class VerisignSpiderMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
if self.detect_bot(request):
# Handle bot traffic
pass
return self.get_response(request)
// JavaScript/Node.js Detection for Verisign Spider
const express = require('express');
const app = express();
// Middleware to detect Verisign Spider
function detectVerisignSpider(req, res, next) {
const userAgent = req.headers['user-agent'] || '';
const pattern = /Verisign Spider/i;
if (pattern.test(userAgent)) {
// Log bot detection
console.log('Verisign Spider detected from IP:', req.ip);
// Set cache headers
res.set({
'Cache-Control': 'public, max-age=3600',
'X-Robots-Tag': 'noarchive'
});
// Mark request as bot
req.isBot = true;
req.botName = 'Verisign Spider';
}
next();
}
app.use(detectVerisignSpider);
# Apache .htaccess rules for Verisign Spider
# Block completely
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Verisign Spider [NC]
RewriteRule .* - [F,L]
# Or redirect to a static version
RewriteCond %{HTTP_USER_AGENT} Verisign Spider [NC]
RewriteCond %{REQUEST_URI} !^/static/
RewriteRule ^(.*)$ /static/$1 [L]
# Or set environment variable for PHP
SetEnvIfNoCase User-Agent "Verisign Spider" is_bot=1
# Add cache headers for this bot
<If "%{HTTP_USER_AGENT} =~ /Verisign Spider/i">
Header set Cache-Control "public, max-age=3600"
Header set X-Robots-Tag "noarchive"
</If>
# Nginx configuration for Verisign Spider
# Map user agent to variable
map $http_user_agent $is_verisign_spider {
default 0;
~*Verisign Spider 1;
}
server {
# Block the bot completely
if ($is_verisign_spider) {
return 403;
}
# Or serve cached content
location / {
if ($is_verisign_spider) {
root /var/www/cached;
try_files $uri $uri.html $uri/index.html @backend;
}
try_files $uri @backend;
}
# Add headers for bot requests
location @backend {
if ($is_verisign_spider) {
add_header Cache-Control "public, max-age=3600";
add_header X-Robots-Tag "noarchive";
}
proxy_pass http://backend;
}
}
Should You Block This Bot?
Recommendations based on your website type:
| Site Type | Recommendation | Reasoning |
|---|---|---|
| E-commerce | Optional | Evaluate based on bandwidth usage vs. benefits |
| Blog/News | Allow | Increases content reach and discoverability |
| SaaS Application | Block | No benefit for application interfaces; preserve resources |
| Documentation | Selective | Allow for public docs, block for internal docs |
| Corporate Site | Limit | Allow for public pages, block sensitive areas like intranets |
Advanced robots.txt Configurations
E-commerce Site Configuration
User-agent: Verisign Spider
Crawl-delay: 5
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /api/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&page=
Allow: /products/
Allow: /categories/
Sitemap: https://example.com/sitemap.xml
Publishing/Blog Configuration
User-agent: Verisign Spider
Crawl-delay: 10
Disallow: /wp-admin/
Disallow: /drafts/
Disallow: /preview/
Disallow: /*?replytocom=
Allow: /
SaaS/Application Configuration
User-agent: Verisign Spider
Disallow: /app/
Disallow: /api/
Disallow: /dashboard/
Disallow: /settings/
Allow: /
Allow: /pricing/
Allow: /features/
Allow: /docs/
Quick Reference
User Agent Match
Verisign Spider
Robots.txt Name
Verisign Spider
Category
security, other
Respects robots.txt
Yes
Copied to clipboard!