AI crawlers and your robots.txt, without the myths
Blocking every “AI bot” can quietly remove you from AI search answers. Blocking only the training crawlers is the opt-out most businesses actually want. The crawlers, what each one does, and a decision tree for your robots.txt.
A common reaction to AI search is to block “the AI bots” in robots.txt. It feels protective. But each major AI company runs several crawlers with different jobs, and robots.txt treats each one separately. A single broad block can switch off the crawlers that put your pages into AI search answers. A targeted block on the training crawlers is usually what people actually meant.
Three jobs, many names
The crawlers, by job, as documented by each company
| Company | Training | Search | User request |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | No training crawler listed | PerplexityBot | Perplexity-User |
| Google-Extended (a control token, which also covers grounding in the Gemini apps) | Googlebot | User-triggered fetchers | |
| Apple | Applebot-Extended (a control token) | Applebot | – |
What the documentation actually says
- OpenAI: “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” OAI-SearchBot is separate: sites that opt out of it “will not be shown in ChatGPT search answers”, though OpenAI notes they can still appear as navigational links.
- Anthropic: restricting ClaudeBot signals that future material should be excluded from training. Claude-SearchBot and Claude-User serve search and user requests; Anthropic says blocking Claude-User may reduce visibility for user-directed web search.
- Perplexity: PerplexityBot indexes for search results and “is not used to crawl content for AI foundation models”. Perplexity-User, acting for a person, “generally ignores robots.txt rules”.
- Google: Google-Extended controls whether Google may use your content to train future Gemini models and to ground answers in the Gemini apps. It “does not impact a site’s inclusion in Google Search”. Google’s user-triggered fetchers “generally ignore robots.txt rules” because a person asked for the page.
- Apple: disallowing Applebot-Extended opts your content out of training Apple’s foundation models, while it “will remain discoverable through Spotlight, Siri, and Safari”.
A decision tree for your file
Example: appear in AI search, opt out of OpenAI, Anthropic and Apple training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Google-Extended is not listed: blocking it would also
# keep your pages out of Gemini app answers.
# Search crawlers and user-request agents are not listed,
# so they follow your normal rules.No special AI files required
Google’s guidance is direct: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” For Google, being crawlable, indexable and useful still does the work. A shared, machine-readable “no AI training” preference is also on the way: Google says it intends to support one in early 2027. Until then, the per-crawler rules above are how you express it.
To see which AI crawlers actually visit your site, look at your server logs or a tool with agent analytics. Our piece on AI agent traffic explains what those numbers can and can’t show.
Sources
- OpenAI: Overview of OpenAI crawlers viewed 11 Nov 2026
- Anthropic: Does Anthropic crawl data from the web? viewed 11 Nov 2026
- Perplexity: Perplexity crawlers viewed 11 Nov 2026
- Google: Google’s common crawlers (Google-Extended) viewed 11 Nov 2026
- Google: User-triggered fetchers viewed 11 Nov 2026
- Apple: About Applebot viewed 11 Nov 2026
- Google Search Central: AI features and your website viewed 11 Nov 2026
Directory figures come from Hyperfind’s profiles of 34 tools, not the whole market. Prices follow each provider’s published offer on the date shown in its profile.