YandexRenderResourcesBot
Yandex •
Web Crawler
What is YandexRenderResourcesBot?
Yandex documents this as a resource-fetching bot for pages it renders with JavaScript, separate from its general indexing crawler. Its webmaster documentation says this bot may ignore a resource's own robots.txt restriction if the HTML page using that resource is accessible; it does not fetch resources for disallowed HTML pages. That distinction matters when interpreting requests or writing robots rules.
User Agent String
Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0
How to Control YandexRenderResourcesBot
Block Completely
To prevent YandexRenderResourcesBot from accessing your entire website, add this to your robots.txt file:
# Block YandexRenderResourcesBot
User-agent: YandexRenderResourcesBot
Disallow: /
Block Specific Directories
To restrict access to certain parts of your site while allowing others:
User-agent: YandexRenderResourcesBot
Disallow: /admin/
Disallow: /private/
Disallow: /wp-admin/
Allow: /public/
Set Crawl Delay
To slow down the crawl rate (note: not all bots respect this directive):
User-agent: YandexRenderResourcesBot
Crawl-delay: 10
How to Verify YandexRenderResourcesBot
Verification Method:
Check the source IP with Yandex's robot verification guidance; a user-agent string is not sufficient.
Check the source IP with Yandex's robot verification guidance; a user-agent string is not sufficient.
Learn more in the official documentation.
Detection Patterns
Multiple ways to detect YandexRenderResourcesBot in your application:
Basic Pattern
/YandexRenderResourcesBot/i
Strict Pattern
/^Mozilla\/5\.0 \(compatible; YandexRenderResourcesBot\/1\.0; \+http\:\/\/yandex\.com\/bots\) AppleWebKit\/537\.36 \(KHTML, like Gecko\) Chrome\/108\.0\.0\.0$/
Flexible Pattern
/YandexRenderResourcesBot[\s\/]?[\d\.]*?/i
Vendor Match
/.*Yandex.*YandexRenderResourcesBot/i
Implementation Examples
// PHP Detection for YandexRenderResourcesBot
function detect_yandexrenderresourcesbot() {
$user_agent = $_SERVER['HTTP_USER_AGENT'] ?? '';
$pattern = '/YandexRenderResourcesBot/i';
if (preg_match($pattern, $user_agent)) {
// Log the detection
error_log('YandexRenderResourcesBot detected from IP: ' . $_SERVER['REMOTE_ADDR']);
// Set cache headers
header('Cache-Control: public, max-age=3600');
header('X-Robots-Tag: noarchive');
// Optional: Serve cached version
if (file_exists('cache/' . md5($_SERVER['REQUEST_URI']) . '.html')) {
readfile('cache/' . md5($_SERVER['REQUEST_URI']) . '.html');
exit;
}
return true;
}
return false;
}
# Python/Flask Detection for YandexRenderResourcesBot
import re
from flask import request, make_response
def detect_yandexrenderresourcesbot():
user_agent = request.headers.get('User-Agent', '')
pattern = r'YandexRenderResourcesBot'
if re.search(pattern, user_agent, re.IGNORECASE):
# Create response with caching
response = make_response()
response.headers['Cache-Control'] = 'public, max-age=3600'
response.headers['X-Robots-Tag'] = 'noarchive'
return True
return False
# Django Middleware
class yandexrenderresourcesbotMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
if self.detect_bot(request):
# Handle bot traffic
pass
return self.get_response(request)
// JavaScript/Node.js Detection for YandexRenderResourcesBot
const express = require('express');
const app = express();
// Middleware to detect YandexRenderResourcesBot
function detect_yandexrenderresourcesbot(req, res, next) {
const userAgent = req.headers['user-agent'] || '';
const pattern = /YandexRenderResourcesBot/i;
if (pattern.test(userAgent)) {
// Log bot detection
console.log('YandexRenderResourcesBot detected from IP:', req.ip);
// Set cache headers
res.set({
'Cache-Control': 'public, max-age=3600',
'X-Robots-Tag': 'noarchive'
});
// Mark request as bot
req.isBot = true;
req.botName = 'YandexRenderResourcesBot';
}
next();
}
app.use(detect_yandexrenderresourcesbot);
# Apache .htaccess rules for YandexRenderResourcesBot
# Block completely
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} YandexRenderResourcesBot [NC]
RewriteRule .* - [F,L]
# Or redirect to a static version
RewriteCond %{HTTP_USER_AGENT} YandexRenderResourcesBot [NC]
RewriteCond %{REQUEST_URI} !^/static/
RewriteRule ^(.*)$ /static/$1 [L]
# Or set environment variable for PHP
SetEnvIfNoCase User-Agent "YandexRenderResourcesBot" is_bot=1
# Add cache headers for this bot
<If "%{HTTP_USER_AGENT} =~ /YandexRenderResourcesBot/i">
Header set Cache-Control "public, max-age=3600"
Header set X-Robots-Tag "noarchive"
</If>
# Nginx configuration for YandexRenderResourcesBot
# Map user agent to variable
map $http_user_agent $is_yandexrenderresourcesbot {
default 0;
~*YandexRenderResourcesBot 1;
}
server {
# Block the bot completely
if ($is_yandexrenderresourcesbot) {
return 403;
}
# Or serve cached content
location / {
if ($is_yandexrenderresourcesbot) {
root /var/www/cached;
try_files $uri $uri.html $uri/index.html @backend;
}
try_files $uri @backend;
}
# Add headers for bot requests
location @backend {
if ($is_yandexrenderresourcesbot) {
add_header Cache-Control "public, max-age=3600";
add_header X-Robots-Tag "noarchive";
}
proxy_pass http://backend;
}
}
Should You Block This Bot?
Recommendations based on your website type:
| Site Type | Recommendation | Reasoning |
|---|---|---|
| E-commerce | Allow | Essential for product visibility in search results |
| Blog/News | Allow | Increases content reach and discoverability |
| SaaS Application | Block | No benefit for application interfaces; preserve resources |
| Documentation | Allow | Improves documentation discoverability for developers |
| Corporate Site | Allow | Allow for public pages, block sensitive areas like intranets |
Advanced robots.txt Configurations
E-commerce Site Configuration
User-agent: YandexRenderResourcesBot
Crawl-delay: 5
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /api/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&page=
Allow: /products/
Allow: /categories/
Sitemap: https://example.com/sitemap.xml
Publishing/Blog Configuration
User-agent: YandexRenderResourcesBot
Crawl-delay: 10
Disallow: /wp-admin/
Disallow: /drafts/
Disallow: /preview/
Disallow: /*?replytocom=
Allow: /
SaaS/Application Configuration
User-agent: YandexRenderResourcesBot
Disallow: /app/
Disallow: /api/
Disallow: /dashboard/
Disallow: /settings/
Allow: /
Allow: /pricing/
Allow: /features/
Allow: /docs/
Quick Reference
User Agent Match
YandexRenderResourcesBot
Robots.txt Name
YandexRenderResourcesBot
Category
search
Respects robots.txt
May not respect
Copied to clipboard!