Go-http-client
Go •
Since 2009
Quick Actions
Official Docs
What is Go-http-client?
Go-http-client is the default user agent for Go's net/http package when no custom user agent is specified. As Go has become popular for building web services, microservices, and cloud-native applications, this user agent frequently appears in server logs. It's commonly seen from monitoring services, API clients, and various automation tools built with Go. The efficiency and performance of Go makes it a popular choice for high-performance web clients and services, contributing to the prevalence of this user agent.
User Agent String
Go-http-client/1.1
How to Control Go-http-client
Block Completely
To prevent Go-http-client from accessing your entire website, add this to your robots.txt file:
# Block Go-http-client
User-agent: Go-http-client
Disallow: /
Block Specific Directories
To restrict access to certain parts of your site while allowing others:
User-agent: Go-http-client
Disallow: /admin/
Disallow: /private/
Disallow: /wp-admin/
Allow: /public/
Set Crawl Delay
To slow down the crawl rate (note: not all bots respect this directive):
User-agent: Go-http-client
Crawl-delay: 10
How to Verify Go-http-client
Verification Method:
Default Go HTTP client user agent
Default Go HTTP client user agent
Learn more in the official documentation.
Detection Patterns
Multiple ways to detect Go-http-client in your application:
Basic Pattern
/Go\-http\-client/iStrict Pattern
/^Go\-http\-client/1\.1$/Flexible Pattern
/Go\-http\-client[\s\/]?[\d\.]*?/iVendor Match
/.*Go.*Go\-http\-client/iImplementation Examples
// PHP Detection for Go-http-client
function detect_go_http_client() {
$user_agent = $_SERVER['HTTP_USER_AGENT'] ?? '';
$pattern = '/Go\\-http\\-client/i';
if (preg_match($pattern, $user_agent)) {
// Log the detection
error_log('Go-http-client detected from IP: ' . $_SERVER['REMOTE_ADDR']);
// Set cache headers
header('Cache-Control: public, max-age=3600');
header('X-Robots-Tag: noarchive');
// Optional: Serve cached version
if (file_exists('cache/' . md5($_SERVER['REQUEST_URI']) . '.html')) {
readfile('cache/' . md5($_SERVER['REQUEST_URI']) . '.html');
exit;
}
return true;
}
return false;
}
# Python/Flask Detection for Go-http-client
import re
from flask import request, make_responsedef detect_go_http_client():
user_agent = request.headers.get('User-Agent', '')
pattern = r'Go-http-client'
if re.search(pattern, user_agent, re.IGNORECASE):
# Create response with caching
response = make_response()
response.headers['Cache-Control'] = 'public, max-age=3600'
response.headers['X-Robots-Tag'] = 'noarchive'
return True
return False# Django Middleware
class GohttpclientMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
if self.detect_bot(request):
# Handle bot traffic
pass
return self.get_response(request)
// JavaScript/Node.js Detection for Go-http-client
const express = require('express');
const app = express();// Middleware to detect Go-http-client
function detectGohttpclient(req, res, next) {
const userAgent = req.headers['user-agent'] || '';
const pattern = /Go-http-client/i;
if (pattern.test(userAgent)) {
// Log bot detection
console.log('Go-http-client detected from IP:', req.ip);
// Set cache headers
res.set({
'Cache-Control': 'public, max-age=3600',
'X-Robots-Tag': 'noarchive'
});
// Mark request as bot
req.isBot = true;
req.botName = 'Go-http-client';
}
next();
}app.use(detectGohttpclient);
# Apache .htaccess rules for Go-http-client# Block completely
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Go\-http\-client [NC]
RewriteRule .* - [F,L]# Or redirect to a static version
RewriteCond %{HTTP_USER_AGENT} Go\-http\-client [NC]
RewriteCond %{REQUEST_URI} !^/static/
RewriteRule ^(.*)$ /static/$1 [L]# Or set environment variable for PHP
SetEnvIfNoCase User-Agent "Go\-http\-client" is_bot=1# Add cache headers for this bot
<If "%{HTTP_USER_AGENT} =~ /Go\-http\-client/i">
Header set Cache-Control "public, max-age=3600"
Header set X-Robots-Tag "noarchive"
</If>
# Nginx configuration for Go-http-client# Map user agent to variable
map $http_user_agent $is_go_http_client {
default 0;
~*Go\-http\-client 1;
}server {
# Block the bot completely
if ($is_go_http_client) {
return 403;
}
# Or serve cached content
location / {
if ($is_go_http_client) {
root /var/www/cached;
try_files $uri $uri.html $uri/index.html @backend;
}
try_files $uri @backend;
}
# Add headers for bot requests
location @backend {
if ($is_go_http_client) {
add_header Cache-Control "public, max-age=3600";
add_header X-Robots-Tag "noarchive";
}
proxy_pass http://backend;
}
}
Should You Block This Bot?
Recommendations based on your website type:
| Site Type | Recommendation | Reasoning |
|---|---|---|
| E-commerce | Optional | Evaluate based on bandwidth usage vs. benefits |
| Blog/News | Allow | Increases content reach and discoverability |
| SaaS Application | Block | No benefit for application interfaces; preserve resources |
| Documentation | Selective | Allow for public docs, block for internal docs |
| Corporate Site | Limit | Allow for public pages, block sensitive areas like intranets |
Advanced robots.txt Configurations
E-commerce Site Configuration
User-agent: Go-http-client
Crawl-delay: 5
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /api/
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&page=
Allow: /products/
Allow: /categories/
Sitemap: https://example.com/sitemap.xml
Publishing/Blog Configuration
User-agent: Go-http-client
Crawl-delay: 10
Disallow: /wp-admin/
Disallow: /drafts/
Disallow: /preview/
Disallow: /*?replytocom=
Allow: /
SaaS/Application Configuration
User-agent: Go-http-client
Disallow: /app/
Disallow: /api/
Disallow: /dashboard/
Disallow: /settings/
Allow: /
Allow: /pricing/
Allow: /features/
Allow: /docs/
Quick Reference
User Agent Match
Go-http-clientRobots.txt Name
Go-http-clientCategory
otherRespects robots.txt
May not respect
Copied to clipboard!