AI Crawler Checker

Paste your robots.txt and see which AI crawlers it lets in — and which of the blocked ones were the ones that could have cited you.

Paste the whole file, comments and all. Fetch yours from yourdomain.com/robots.txt.
The site root is always tested as well. Use a real article path — a rule that blocks a whole section is easy to miss when you only test the home page.
Crawler What it is for Blocking it costs you Your file says
GPTBotOpenAI TrainingCollects pages to train future OpenAI models. It is not the agent that fetches a page to answer a question. Nothing in citations. Blocking it keeps your text out of future training runs and does not affect whether ChatGPT can quote you today.
ChatGPT-UserOpenAI Live retrievalFetches a page during a conversation, because something a user asked sent ChatGPT to go and look at it. Citations. Block this and the page cannot be opened, summarised or linked while someone is asking about your topic.
OAI-SearchBotOpenAI Search indexBuilds the index behind ChatGPT's search results. OpenAI documents it as separate from training. Citations. This is the crawler that decides whether you appear in ChatGPT search at all.
ClaudeBotAnthropic TrainingAnthropic's general crawler, documented as collecting data for model training. Nothing in citations, on the same logic as GPTBot.
Claude-UserAnthropic Live retrievalFetches a specific page because a person using Claude asked for it. Citations, in the same way as ChatGPT-User.
Claude-SearchBotAnthropic Search indexCrawls to support the search results Claude draws on and cites. Citations.
anthropic-aiAnthropic — legacy token TrainingAn older token that appears in thousands of robots.txt files. Anthropic's documented agents are the three above; this one is kept here because your file probably mentions it. Nothing either way. Leaving the line in costs nothing and removing it gains nothing.
PerplexityBotPerplexity Search indexIndexes pages so they can be listed and cited in Perplexity answers. Perplexity documents it as not used for training. Citations, and directly. Perplexity puts numbered sources next to every claim; blocking this removes you from that list.
Perplexity-UserPerplexity Live retrievalFetches a page because a user's question sent Perplexity to it. Perplexity's own documentation says fetches a person asked for may not follow robots.txt, so a rule here may simply not be obeyed. Citations, where the rule is honoured at all.
Google-ExtendedGoogle Training and groundingNot a crawler. A token that controls whether pages Googlebot already fetched may be used to train and to ground Gemini. It has no effect on Search. Some. It does not touch Search or AI Overviews, but it does cover grounding in the Gemini apps, so blocking it can take you out of answers there.
GooglebotGoogle Search indexThe Search crawler. AI Overviews and AI Mode are built on the Search index, so this one governs those too. Everything. There is no separate opt-out for AI Overviews: the only controls are nosnippet, max-snippet and data-nosnippet, which limit how much text can be shown rather than whether you appear.
bingbotMicrosoft Search indexThe Bing crawler. Copilot's answers draw on the Bing index. Bing and Copilot together. People block bingbot to save crawl budget and lose both at once.
ApplebotApple Search indexPowers Siri and Spotlight suggestions, and the search behind Apple's own surfaces. Visibility inside Apple products.
Applebot-ExtendedApple TrainingNot a crawler either: the token that says whether pages Applebot already fetched may train Apple's foundation models. Nothing in search. Disallowing it leaves Applebot's indexing untouched.
Meta-ExternalAgentMeta TrainingMeta's crawler for training data and for improving its products. Nothing in citations.
Meta-ExternalFetcherMeta Live retrievalFetches a specific page on behalf of a Meta AI user who asked about it. Citations inside Meta AI. The pair is the clearest illustration of the split: same company, two tokens, two completely different decisions.
AmazonbotAmazon Search indexCrawls the web to answer questions through Alexa and to support Amazon's own services. Answers on Amazon's surfaces.
BytespiderByteDance TrainingCollects data for ByteDance's models. Widely reported by site owners as heavy on crawl rate, which is an observation rather than a published figure. Little for most sites outside its markets. This is usually a bandwidth decision, not a visibility one.
CCBotCommon Crawl Public datasetBuilds the Common Crawl archive, a free public crawl of the web that many models have been trained on. Nothing in citations. Note that blocking it only affects future crawls — whatever is already in the archive stays there.
cohere-aiCohere TrainingA token that appears in most blocklists. Cohere's use of it is not clearly documented, so read this row as informed guesswork rather than fact. Probably nothing. Included because your file likely names it.
DiffbotDiffbot Commercial datasetCrawls to build a knowledge graph sold to other companies, some of whom use it for AI. Nothing directly. It can feed the entity data some tools resolve brands against, which is a second-order argument for leaving it alone.
TimpibotTimpi Search indexCrawls for Timpi, a distributed search index. Very little today. A small index.
omgiliWebz.io Commercial datasetCollects forum and news content for datasets Webz.io sells, some of which end up in AI training sets. Nothing in citations.
DuckAssistBotDuckDuckGo Live retrievalFetches pages for DuckDuckGo's AI-assisted answers. Citations in DuckDuckGo's answer box.

This catalogue was compiled in September 2026. Tokens get renamed, split and retired — before you act on a row, check it against the operator's own crawler documentation. The purpose and cost columns are a judgement about what each bot is for, not a quotation from anyone.

Nothing you type here is uploaded, stored or sent anywhere. It all runs in your browser.

Training and retrieval are two different decisions

Almost every argument about blocking AI crawlers collapses because the two sides are talking about different bots. There are, roughly, three jobs:

  • Training crawlers collect text to build the next model. GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot. Blocking them is a licensing decision. It does not remove you from anything a user sees today.
  • Retrieval agents fetch one page, now, because someone asked a question it might answer. ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher. Blocking these takes you out of answers that would have linked to you.
  • Search crawlers build the index those answers are drawn from. OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, bingbot. Blocking these is the same decision as blocking a search engine, which is to say it is almost never what you meant.

The mistake we see most often is a single copied block that disallows twenty tokens at once — lifted from a blog post, never read — which keeps you out of AI answers while the training corpora that already contain your 2023 archive are entirely unaffected. If you want to be cited and not trained, you have to separate the two, and the operators that offer that separation have published the tokens to do it with.

How this tool decides

It implements the matching rules from RFC 9309, the same ones a search engine uses. Three of them catch people out:

  • Groups are not inherited. If a crawler has its own User-agent block, the rules under User-agent: * do not apply to it at all — not in addition, not as a fallback. A group of User-agent: GPTBot with an empty Disallow: gives GPTBot more access than everyone else, including your admin paths.
  • The longest matching rule wins, not the first. Disallow: /blog/ and Allow: /blog/public/ together permit the public folder, whichever order they are written in.
  • Matching is by token, not by substring. User-agent: Claude does not match ClaudeBot. Tokens are matched whole and case-insensitively, so a near-miss is a rule that quietly does nothing.

What robots.txt cannot do

It is a request, honoured voluntarily. The bots in the table above are the ones that publish a token and, as far as anyone can observe, respect it — that is precisely why they are in a table. Scrapers that do not identify themselves are a different problem with a different answer, and that answer is at the server or the CDN, not in this file. Blocking a crawler also does nothing about content of yours that is quoted, syndicated or reposted elsewhere: the copy on someone else's domain is governed by their robots.txt, not yours.

Then what?

Once you know what you are allowing, use the AI Crawler Control Generator to write the rules you actually meant, and the AI Search Readiness Checker to see whether the pages you are letting them reach are written in a way anything can quote.

Questions people ask

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot is OpenAI’s training crawler. What fetches your page during a conversation is ChatGPT-User, and what indexes it for ChatGPT search is OAI-SearchBot. Blocking GPTBot alone leaves both of those working.

Can I block AI Overviews without leaving Google Search?

Not by presence. Google-Extended covers Gemini, not Search, and AI Overviews are generated from the Search index, so the only way to be absent from them is to be absent from Search. The partial controls are nosnippet, max-snippet and data-nosnippet, which limit how much of your text can be displayed.

Why does this tool say a bot is allowed when my file lists it?

Usually one of three things: the token is spelled differently from the published one, the rule sits above the User-agent line instead of below it, or the crawler has its own group and the Disallow you meant for it is sitting in the group for everyone else. The matched rule is shown next to each verdict so you can see which.

Is a blocked AI crawler the same as a noindex?

No, and confusing the two is how sites disappear. robots.txt stops the fetch; noindex stops the indexing and requires the fetch to happen so the directive can be read. A page blocked in robots.txt can still be listed with no description, because the block prevented the crawler from ever seeing the noindex.

Is my robots.txt uploaded anywhere when I paste it?

No. The parsing and matching run in your browser. This page has no backend and makes no network request with anything you type.

Related tools

Being crawlable is the floor, not the ceiling

Letting the right bots in only makes you eligible to be quoted. What makes a model treat you as worth quoting is the same thing that always mattered: other people referencing you.

Book a Call More free tools