Cohere-Ai

Name: Cohere-Ai
Author: Cohere

Cohere • Since 2022

Ai Respects robots.txt

#ai #nlp #training #crawler

Quick Actions

Official Docs

What is Cohere-Ai?

Cohere-AI is the web crawler operated by Cohere, a leading AI company specializing in large language models for enterprise use. The bot collects training data to improve Cohere's natural language processing models, which power various enterprise AI applications. Cohere focuses on providing accessible, customizable AI models for businesses, and their crawler helps ensure these models understand diverse content and use cases. As enterprise AI adoption accelerates, Cohere's crawler plays a vital role in building models that can understand and generate human-like text for business applications.

User Agent String

Mozilla/5.0 (compatible; Cohere-AI/1.0; +https://cohere.com/)

How to Control Cohere-Ai

Block Completely

To prevent Cohere-Ai from accessing your entire website, add this to your robots.txt file:

# Block Cohere-Ai
User-agent: Cohere-Ai
Disallow: /

Block Specific Directories

To restrict access to certain parts of your site while allowing others:

User-agent: Cohere-Ai
Disallow: /admin/
Disallow: /private/
Disallow: /wp-admin/
Allow: /public/

Set Crawl Delay

To slow down the crawl rate (note: not all bots respect this directive):

User-agent: Cohere-Ai
Crawl-delay: 10

How to Verify Cohere-Ai

Verification Method:
Verify with Cohere's published information

Learn more in the official documentation.

⚠️ AI Training Notice
This bot may collect and use your website content for AI model training. Consider whether you want your content used for this purpose before allowing access.

Detection Patterns

Multiple ways to detect Cohere-Ai in your application:

Basic Pattern

/Cohere\-Ai/i

Strict Pattern

/^Mozilla/5\.0 $compatible; Cohere\-AI/1\.0; \+https\://cohere\.com/$$/

Flexible Pattern

/Cohere\-Ai[\s\/]?[\d\.]*?/i

Vendor Match

/.*Cohere.*Cohere\-Ai/i

Implementation Examples

// PHP Detection for Cohere-Ai function detect_cohere_ai() { $user_agent = $_SERVER['HTTP_USER_AGENT'] ?? ''; $pattern = '/Cohere\\-Ai/i'; if (preg_match($pattern, $user_agent)) { // Log the detection error_log('Cohere-Ai detected from IP: ' . $_SERVER['REMOTE_ADDR']); // Set cache headers header('Cache-Control: public, max-age=3600'); header('X-Robots-Tag: noarchive'); // Optional: Serve cached version if (file_exists('cache/' . md5($_SERVER['REQUEST_URI']) . '.html')) { readfile('cache/' . md5($_SERVER['REQUEST_URI']) . '.html'); exit; } return true; } return false; }

# Python/Flask Detection for Cohere-Ai import re from flask import request, make_responsedef detect_cohere_ai(): user_agent = request.headers.get('User-Agent', '') pattern = r'Cohere-Ai' if re.search(pattern, user_agent, re.IGNORECASE): # Create response with caching response = make_response() response.headers['Cache-Control'] = 'public, max-age=3600' response.headers['X-Robots-Tag'] = 'noarchive' return True return False# Django Middleware class CohereAiMiddleware: def __init__(self, get_response): self.get_response = get_response def __call__(self, request): if self.detect_bot(request): # Handle bot traffic pass return self.get_response(request)

// JavaScript/Node.js Detection for Cohere-Ai const express = require('express'); const app = express();// Middleware to detect Cohere-Ai function detectCohereAi(req, res, next) { const userAgent = req.headers['user-agent'] || ''; const pattern = /Cohere-Ai/i; if (pattern.test(userAgent)) { // Log bot detection console.log('Cohere-Ai detected from IP:', req.ip); // Set cache headers res.set({ 'Cache-Control': 'public, max-age=3600', 'X-Robots-Tag': 'noarchive' }); // Mark request as bot req.isBot = true; req.botName = 'Cohere-Ai'; } next(); }app.use(detectCohereAi);

# Apache .htaccess rules for Cohere-Ai# Block completely RewriteEngine On RewriteCond %{HTTP_USER_AGENT} Cohere\-Ai [NC] RewriteRule .* - [F,L]# Or redirect to a static version RewriteCond %{HTTP_USER_AGENT} Cohere\-Ai [NC] RewriteCond %{REQUEST_URI} !^/static/ RewriteRule ^(.*)$ /static/$1 [L]# Or set environment variable for PHP SetEnvIfNoCase User-Agent "Cohere\-Ai" is_bot=1# Add cache headers for this bot <If "%{HTTP_USER_AGENT} =~ /Cohere\-Ai/i"> Header set Cache-Control "public, max-age=3600" Header set X-Robots-Tag "noarchive" </If>

# Nginx configuration for Cohere-Ai# Map user agent to variable map $http_user_agent $is_cohere_ai { default 0; ~*Cohere\-Ai 1; }server { # Block the bot completely if ($is_cohere_ai) { return 403; } # Or serve cached content location / { if ($is_cohere_ai) { root /var/www/cached; try_files $uri $uri.html $uri/index.html @backend; } try_files $uri @backend; } # Add headers for bot requests location @backend { if ($is_cohere_ai) { add_header Cache-Control "public, max-age=3600"; add_header X-Robots-Tag "noarchive"; } proxy_pass http://backend; } }

Should You Block This Bot?

Recommendations based on your website type:

Site Type	Recommendation	Reasoning
E-commerce	Limit Access	Protect pricing and inventory data from AI training
Blog/News	Consider Blocking	Your content may be used for AI training without compensation
SaaS Application	Block	No benefit for application interfaces; preserve resources
Documentation	Selective	Allow for public docs, block for internal docs
Corporate Site	Limit	Allow for public pages, block sensitive areas like intranets

Advanced robots.txt Configurations

E-commerce Site Configuration

User-agent: Cohere-Ai Crawl-delay: 5 Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /api/ Disallow: /*?sort= Disallow: /*?filter= Disallow: /*&page= Allow: /products/ Allow: /categories/ Sitemap: https://example.com/sitemap.xml

Publishing/Blog Configuration

User-agent: Cohere-Ai # Blocking AI training bot Disallow: /

SaaS/Application Configuration

User-agent: Cohere-Ai Disallow: /app/ Disallow: /api/ Disallow: /dashboard/ Disallow: /settings/ Allow: / Allow: /pricing/ Allow: /features/ Allow: /docs/

Quick Reference

User Agent Match

Cohere-Ai

Robots.txt Name