Allow or block AI crawlers?
Updated: 2026-11
Anyone who wants to be mentioned in AI answers should allow the crawlers of AI search services. Blocking can make sense for content that should not be used to train models. Because many providers run separate crawlers for training, search and user-triggered fetches, control can be differentiated rather than all or nothing.
Since AI systems started reading web content, many site owners ask whether to allow it. The answer depends on the goal: visibility in AI answers or control over whether your content trains models.
One distinction is often overlooked: several providers separate their crawlers by purpose. OpenAI, for example, names GPTBot for training and OAI-SearchBot for search; Google uses the Google-Extended token to control use for Gemini, without affecting Google Search or AI Overviews, which run on the regular Googlebot.
Head to head
| Criterion | Allow AI crawlers | Block AI crawlers |
|---|---|---|
| Visibility in AI answers | Possible if search crawlers may fetch pages. | Limited or impossible for blocked services. |
| Use for training | Content may enter training data. | Blocked training crawlers generally respect it. |
| Google Search | Unchanged. | Unchanged as long as only Google-Extended is blocked, not Googlebot. |
| Referral traffic | Possible from ChatGPT, Perplexity and others. | Lost for blocked services. |
| Enforcement | None needed. | robots.txt is a request; not all crawlers comply. |
When Allow AI crawlers wins
- —The business model lives on visibility and enquiries, as for service providers and software vendors.
- —Content is public anyway and should be cited as a source.
- —You want to appear in AI answers to category and problem questions.
When Block AI crawlers wins
- —Content is the product itself, as for publishers, databases or paid expert content.
- —You want to exclude training use but allow search; then only training crawlers are blocked.
- —Certain areas such as internal documents or customer data should be excluded; they belong behind a login anyway.
Our take
For most companies selling services or software, allowing is the right default, complemented by a deliberate decision per training crawler.
Anyone who blocks should measure the effect: if the company then appears less in AI answers, that is the price of the decision. Our article on AI visibility describes regular measurement.
Parent service: SEO & Organic Growth
Matching offers
AI visibility monitoring
We measure monthly whether and how AI systems mention your company: a fixed panel of brand, category, problem and comparison questions in ChatGPT, Perplexity, Copilot, Gemini and AI Overviews, with several runs, analysis of mentions, citations and errors, and an action plan for content, entities and technology.
GEO/AEO optimisation (AI search)
GEO and AEO optimisation makes your content citable for AI answer engines: structured data, snippet-ready answers, a clean llms.txt and clear entity signals. The result is visibility where search increasingly happens — in the answers of ChatGPT, Perplexity and AI search.
Technical SEO audit
A technical SEO audit examines the foundation of your findability: indexing, Core Web Vitals, structured data, crawling and site architecture. The result is a prioritised list of actions showing which technical levers deliver the greatest visibility gain — and in what order.
FAQ
Will I lose Google rankings if I block Google-Extended?
According to Google, no. Google-Extended concerns use for Gemini, not indexing for Search.
Is robots.txt enough protection?
Reputable crawlers comply, but it is not a technical barrier. What must truly be protected belongs behind authentication.
What does llms.txt do?
An llms.txt can give AI systems a curated overview. It replaces neither robots.txt nor good content, and providers do not guarantee to use it.