Which crawler feeds which product
| Crawler | Company | What it feeds | Does blocking it remove you from answers? |
|---|---|---|---|
GPTBot | OpenAI | Training data for OpenAI models | No. Blocking it keeps your pages out of future training, not out of ChatGPT search answers. |
OAI-SearchBot | OpenAI | ChatGPT search results and citations | Yes. Block it and ChatGPT search cannot read or cite your pages. |
ChatGPT-User | OpenAI | Live fetches when a user asks ChatGPT to open a page | Partly. Blocks on-demand reads of your pages inside a conversation. |
PerplexityBot | Perplexity | Perplexity's index and citations | Yes. Block it and Perplexity cannot cite you. |
Google-Extended | Training and grounding for Gemini | Not for AI Overviews, which use the normal Googlebot. It affects Gemini's use of your content. | |
ClaudeBot | Anthropic | Training data for Anthropic models | No. Training only; Claude's search citations are governed by Claude-SearchBot. |
Claude-SearchBot | Anthropic | Claude's search results and citations | Yes. Block it and Claude's search features cannot cite your pages. |
Applebot-Extended | Apple | Opt-out for Apple Intelligence training | No. It does not crawl; Siri, Spotlight and Safari follow Applebot. |
CCBot | Common Crawl | The open crawl many models train on | Indirectly. Many smaller models learn about you through it. |
Bytespider | ByteDance | Training data for ByteDance models | Only for those models. |
The list is maintained against the vendors' own documentation and was last checked in September 2026. If a crawler you see in your logs is missing, it is either a scraper using a made-up name or newer than this page.
Blocking training is not the same as blocking search
The rule most guides miss. OpenAI runs GPTBot for training and OAI-SearchBot for ChatGPT search; Anthropic runs ClaudeBot for training and Claude-SearchBot for Claude's search citations; you can refuse one and allow the other. Google's AI Overviews are built from the ordinary Google index, which Googlebot crawls, and Google-Extended only controls whether Gemini may use your content. Blocking Google-Extended does not take you out of AI Overviews; blocking Googlebot takes you out of Google. Applebot-Extended is an opt-out label, not a crawler: Siri and Spotlight follow plain Applebot.
So a site that wants to be cited but not trained on can say exactly that, and this checker reads it back to you crawler by crawler.
What a site-wide block costs
Our report scores whether an engine cites your own pages as a source, and a page the engine's crawler may not read can never be cited. Across the scans we run, the AI crawler check carries the single largest weight of the thirteen technical checks, because a block undoes everything else on the list. It does not make an engine forget you exist; it makes the engine describe you only from what other sites say.
How to allow or block each one
A group naming the crawler overrides the * group entirely, so the order is: specific groups first, then the wildcard.
# allow the search crawlers to read and cite
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# refuse training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /admin/Questions about AI crawlers
Does robots.txt stop ChatGPT quoting me?
If OAI-SearchBot is blocked site-wide, ChatGPT search cannot read your pages, so it cannot cite them; it may still name you from what other sites say. Blocking GPTBot alone affects training, not search.
Does Cloudflare's AI bot setting override this?
Cloudflare's block acts before robots.txt is read, so a site can have a permissive robots.txt and still refuse every AI crawler at the edge. This tool reads robots.txt only; if the row says allowed and you use Cloudflare's AI bot blocking, the edge setting wins.
What about Bing and Copilot?
Copilot uses Bing's index, which the normal Bingbot builds. It is not in this list because blocking it is blocking Bing itself.
How long until a change takes effect?
Each crawler re-reads robots.txt on its own schedule, usually within a day. Citations already made do not disappear; new ones stop or start from the next crawl.
Allowed is not the same as named
The full report asks the engines your buyers' questions and tells you whether you are in the answer.