AI Crawler Control Generator

Pick a stance, adjust it bot by bot, and get the robots.txt block — plus the page-level directives that are actually honoured, and the ones that are not.

Changing this resets the toggles below. Touching a toggle switches you to Custom.
Retrieval and search crawlers Tick to allow. These are the ones that fetch or index a page so it can be linked in an answer — blocking them is what costs you citations.
Training crawlers and opt-out tokens Blocking these keeps your text out of future training runs. None of them affect whether you can be quoted today.
Dataset crawlers These collect content into corpora and knowledge graphs that other people buy or download. Blocking affects future crawls only.
Also include
Used only to fill in the policy URL in the TDM block.
Your rules 

          

          

            
          

          

          
Googlebot and bingbot are not on this list on purpose. Blocking them does not opt you out of AI answers — AI Overviews and Copilot are built on those indexes — it opts you out of search entirely, and no generator should make that a checkbox. To check what your current file already does to every AI crawler, including those two, use the AI Crawler Checker.

          
Nothing you type here is uploaded, stored or sent anywhere. It all runs in your browser.

        

      

    

  


  

    

      
Decide the two questions separately

      
There are only two real questions, and mixing them is how sites end up blocking the wrong half of the internet.

      
May they train on my content? This is a licensing question. The answer costs you nothing in traffic either way, because a training crawler is not what fetches a page to answer somebody's question. If your objection to AI is that you were not asked, this is the lever, and GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and CCBot are most of it.

      
May they fetch my page to answer a question and link to me? This is a distribution question, and for most businesses the answer is obviously yes. ChatGPT-User, OAI-SearchBot, Claude-User, PerplexityBot and their equivalents are the crawlers behind the citation. Blocking those is the AI equivalent of blocking Googlebot to protect your content.


      
What the generated file does and does not do

      
    
        
  • These groups are additions, not a replacement. Paste them alongside what you already have. Do not delete your existing User-agent: * block.
    • 
        
  • A crawler with its own group ignores the star group completely. That is why writing the allowed crawlers out explicitly is off by default: an empty Disallow: under User-agent: ChatGPT-User exempts it from every path rule you wrote for everyone else, including the ones protecting your admin and search pages. If you turn it on, repeat those path rules inside each group.
    • 
        
  • robots.txt is not access control. It is a published request. It works on crawlers that identify themselves and choose to respect it — which is all of the ones in this list, and none of the ones you are actually worried about. Stopping a determined scraper is a job for your CDN.
    • 
        
  • It is retroactive about nothing. Blocking CCBot today does not remove your pages from the archives already published, or from any model already trained on them.
    • 
      


      
Page-level directives, ranked by whether they work

      
nosnippet, max-snippet:0 and the data-nosnippet attribute are documented Google directives and are honoured. They limit how much of a page can be displayed, and because AI Overviews are generated from the Search index, they are the only thing that constrains how much of your text appears in one. The cost is that they constrain your ordinary search snippets too.

      
noai and noimageai came out of the art community and are honoured by a small number of platforms that chose to. The large crawlers do not act on them. They are worth shipping only as a statement of intent, and this page will not pretend otherwise.

      
The TDM Reservation Protocol is a W3C community specification for declaring, in a machine-readable way, that you reserve the text and data mining rights that EU copyright law lets you reserve. Adoption is thin. Whether the declaration carries legal weight in your jurisdiction is a question for a lawyer, and nothing on this page is legal advice.

    

  


  

    

      
Questions people ask

      

        

          
What is the sensible default for a business site?

          
Allow everything that retrieves or indexes, block what only trains, and get on with your work. That is the middle preset. You keep every route by which someone could be sent to you, and you stop donating your archive to the next training run.

        

        

          
Will blocking AI crawlers hurt my Google rankings?

          
Not if you block only the AI tokens. Google-Extended is separate from Googlebot and disallowing it has no effect on Search. The damage happens when a copied block includes Googlebot or bingbot, which is why neither is on this page.

        

        

          
Do I need both robots.txt rules and meta tags?

          
They do different jobs. robots.txt decides whether the page is fetched at all. The meta directives decide what may be displayed once it has been fetched, which means they only take effect on crawlers you let in. If you block a bot in robots.txt it never reads your meta tags.

        

        

          
Where exactly does this block go?

          
In the same robots.txt at the root of your domain, after your existing groups. Order does not matter to the parser — groups are matched by user-agent, not by position — but keeping the AI rules together with a comment makes them survive the next person who edits the file.

        

        

          
Can I allow AI crawlers on some sections and not others?

          
Yes. Each group takes path rules like any other, so you can write Disallow: /members/ under GPTBot and leave the rest open. Remember that the group is matched whole: a bot with its own group does not inherit anything from User-agent: *.

        

      

    

  



  

    

      
Related tools

      
    

  


  

    

      
Open to the right crawlers and still invisible?

      
Access is a permission problem and it takes an afternoon. Being worth quoting is an authority problem, and that is the work I do.

      

        Book a Call
        More free tools