User Agent String Database & Bot Directory

A user agent string is an identifier that web browsers, bots, and crawlers send to web servers with every HTTP request. It tells the server what software is making the request, including the application name, version, and sometimes the operating system. For example, Googlebot identifies itself as "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" so that website owners can recognize it as Google's search engine crawler.

You can identify bots crawling your site by checking your server access logs, which record the user agent string for every request. Look for known bot identifiers like Googlebot, Bingbot, GPTBot, or ClaudeBot. You can also use analytics tools that filter bot traffic, or set up server-side logging to flag requests from known crawler user agent strings. This directory catalogs all major bots to help you identify them.

To block AI crawlers, add disallow rules to your robots.txt file. For example, to block OpenAI's GPTBot, add "User-agent: GPTBot" followed by "Disallow: /" on the next line. You can block multiple AI crawlers by adding separate rules for each bot (GPTBot, ClaudeBot, CCBot, Bytespider, etc.). Note that robots.txt relies on voluntary compliance — most reputable AI companies respect these directives, but it is not a guaranteed enforcement mechanism.

robots.txt is a plain text file placed in the root directory of a website (e.g., example.com/robots.txt) that provides instructions to web crawlers about which pages or sections of the site they are allowed or not allowed to access. It uses a simple syntax with "User-agent" to specify which bot the rule applies to and "Disallow" or "Allow" to define access permissions. While it is an industry standard respected by major search engines and legitimate bots, it is advisory rather than enforceable.

Reputable crawlers from established companies like Google, Microsoft, and most AI companies typically respect robots.txt. You can verify compliance by adding a disallow rule for a specific bot and then monitoring your server logs to see if that bot continues to access blocked URLs. Each bot entry in this directory includes a "respects robots.txt" field to help you understand the expected behavior. You can also use the verification methods listed for each bot to confirm the crawler's identity is genuine.

A web crawler (or spider) systematically browses the web to index content, typically following links from page to page. Search engines like Google use crawlers to discover and index web pages. A web scraper extracts specific data from web pages for a particular purpose such as price monitoring or data analysis. The key difference is intent: crawlers aim to index and discover content broadly, while scrapers target specific data from specific pages. Many AI bots function as crawlers that collect training data at scale.

Google uses specialized crawlers for different purposes: Googlebot handles general web search indexing, Googlebot-Image focuses on image search, Googlebot-Video targets video content, AdsBot-Google checks ad landing page quality, Google-Extended collects data for AI training, and Storebot-Google indexes product and shopping content. Each crawler can be independently controlled via robots.txt, giving website owners granular control over which Google services can access their content.

Blocking all bots is generally not recommended, as it would prevent search engines from indexing your site, remove you from search results, and stop social media platforms from generating link previews. Instead, take a selective approach: allow search engine bots and social media crawlers that benefit your visibility, while blocking unwanted bots such as aggressive scrapers or AI training crawlers if you prefer not to contribute to their datasets. Review each bot's purpose before deciding whether to allow or block it.