Alongside classic search engine crawlers, numerous bots from AI providers now visit websites. Anyone wanting to be cited in AI answers must know which they are and what they do. So must anyone wanting to protect their content.
Three purposes
AI crawlers can be distinguished by purpose. Training crawlers collect content for training future models. Search crawlers build an index from which AI assistants with web search generate answers with sources. User fetches retrieve a page in real time because a user asked about it in the assistant. Many providers use separate user agents for these purposes, so the purposes can be controlled separately.
The key user agents
OpenAI uses GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user fetches. Anthropic uses, among others, ClaudeBot, Claude-User and Claude-SearchBot. Perplexity uses PerplexityBot and Perplexity-User. CCBot collects for the open Common Crawl archive, which feeds many training datasets. Google-Extended is not a separate crawler but a robots.txt token by which websites control whether their content is used for Google’s AI models; indexing for Google Search is unaffected. The list changes; the providers’ documentation is authoritative.
Control via robots.txt
In robots.txt rules can be set per user agent: allow everything, block certain areas or exclude a bot completely. Reputable providers state that they respect these rules; robots.txt cannot technically enforce them. Anyone who must protect content effectively needs access control in addition.
The trade-off for GEO
To appear as a source in AI assistants’ answers, allow at least the search and user-fetch crawlers, otherwise the assistant can neither find nor read the page. Whether to allow training crawlers is a separate decision: training can strengthen a brand’s presence in models but releases content for use you do not control. For a consultancy seeking visibility, the benefit usually prevails. In innopulse.io’s robots.txt the main AI crawlers are therefore addressed by name.
Checking
Server logs show which bots actually visit and how often. Regular review shows whether rules work and whether new bots appear. How to become citable beyond that is covered in our decision guide GEO or classic SEO.