User Agent Directory
A comprehensive database of 124 verified user agents crawling the web. Identify AI bots, SEO crawlers, search engine spiders and understand their behavior patterns.
124
Total User Agents
17
AI Crawlers
36
Search Engines
24
SEO Tools
Found 36 user agents in category "search"
Vendor: OpenAI
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)
Vendor: Perplexity AI
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/bot)
search
Vendor: Google
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
search
Vendor: Microsoft
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
search
Vendor: Microsoft
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
Vendor: DuckDuckGo
Mozilla/5.0 (compatible; DuckAssistBot/1.0; +https://duckduckgo.com/duckassist)
search
Vendor: Baidu
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
search
Vendor: Apple
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
search
Vendor: Huawei
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
search
Vendor: Seznam
Mozilla/5.0 (compatible; SeznamBot/4.0; +http://napoveda.seznam.cz/seznambot-intro/)
search
Vendor: Google
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
search
Vendor: Google
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
search
Vendor: Yandex
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
What are User Agents?
User agents are strings that identify the software making requests to your website. They help servers understand what type of client is accessing the content - whether it's a browser, search engine crawler, SEO tool, or AI bot.
Why This Matters
- Control which bots can access your content
- Identify AI crawlers harvesting data
- Monitor SEO tools analyzing your site
- Understand your traffic sources better
How to Use This Data
- Create robots.txt rules: Block or allow specific bots
- Server configuration: Set up rate limiting for aggressive crawlers
- Analytics filtering: Exclude bot traffic from reports
- Security monitoring: Identify suspicious crawler activity
Tip: Use the Robots.txt Generator to easily create rules for these user agents.