How to Block Bots from Your Website: robots.txt and Firewalls

The fastest way to learn how to block bots from your website is to add targeted disallow rules to your robots.txt file or block malicious IP ranges at your server firewall. For web scrapers and AI training agents, a text rule at the root of your domain tells compliant automated visitors to stop fetching your pages immediately. For aggressive scrapers that ignore site rules, blocking traffic at the DNS or web application firewall level drops their connection before your server spends resources responding.

Which bot blocking method should you choose?

Bots fall into three distinct camps, and each camp requires a different response:

• Good search bots: Crawlers like Googlebot and Bingbot index your content so users can find you in search results. You rarely want to block these completely.
• AI crawlers and data collectors: Automated crawlers run by companies like OpenAI, Anthropic, and Common Crawl scrape text to train machine learning models or power direct answer engines. These crawlers honor robots.txt files and page-level directives. If you want to manage these specific agents without writing syntax from scratch, you can configure your rules with the AI Crawler Control Generator on Links to Wild.
• Spam and scraper bots: Credential stuffers, form spammers, and rogue content scrapers do not read your robots.txt file. They must be dropped at the server, Content Delivery Network (CDN), or application level.

Here is a breakdown of which tool to reach for based on your setup:

Method 1: Generate custom crawler rules for AI scrapers

If your goal is to stop AI agents from ingesting your articles for model training while preserving regular search engine visibility, generating clean directives prevents syntax errors that could accidentally de-index your site.

1. Open the Links to Wild AI Crawler Control Generator.
2. Select your crawler stance. A sensible default for a business site is to allow bots that retrieve or index content for search users, while blocking bots that only train models on your archive.
3. Adjust permissions token by token. You can disallow agents like GPTBot, ClaudeBot, or CCBot while keeping standard search crawlers untouched.
4. Copy the generated robots.txt block and any page-level meta tags you chose.
5. Paste the block into the robots.txt file located at the root directory of your website.

Method 2: Edit your robots.txt manually with a text editor or CMS

You can create and manage crawler instructions directly using any text editor or through the SEO plugins inside your CMS.

1. Open your existing robots.txt file. If you use WordPress, open your SEO plugin settings (such as Yoast or Rank Math) and look for the file editor menu. If you manage files directly, download robots.txt via FTP or SSH from your public_html folder.
2. Declare the user-agent you want to restrict. To target a specific crawler, write:

   User-agent: GPTBot
Disallow: /

3. If you want to permit a crawler on specific paths while locking down others, define specific path lines:

   User-agent: ClaudeBot
Disallow: /members/
Disallow: /checkout/

   Remember that a bot with its own dedicated group does not inherit rules from the general User-agent: * group. The parser matches the whole block for that specific bot token.
4. Save the file as plain text (UTF-8) and upload it to the root of your domain so it resolves at example.com/robots.txt.

Method 3: Block bad bots at the CDN or server level

Because malicious scrapers ignore voluntary directives, you must cut them off before your server processes the request. The easiest way to handle this without slowing down your host is at the CDN level, such as Cloudflare, or directly in your server configuration.

1. Log in to your CDN or DNS management dashboard (for example, Cloudflare).
2. Navigate to the Security or WAF (Web Application Firewall) section.
3. Enable automatic bot protection features. In Cloudflare, look for Bot Fight Mode in the Bots menu, which challenges suspicious visitors with computational challenges.
4. Create a custom firewall rule to block specific user agents or IP addresses. Set the field to User Agent, select contains, enter the identifier of the offending bot, and set the action to Block.
5. If you manage an Apache server directly, add rewrite conditions to your .htaccess file:

   RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (BadBotName|EvilScraper) [NC]
RewriteRule .* - [F,L]

   This returns a 403 Forbidden status code whenever that user agent makes a request.

Common bot blocking mistakes and how to test your setup

Mistakes in crawler control usually happen when site owners accidentally block search engines or mix up how rules interact.

One major mistake is bundling Googlebot into generic disallow commands. Disallowing Google-Extended only affects Gemini training and does not alter your Google Search visibility, according to Google Search Central documentation. However, writing a broad disallow rule that catches Googlebot or bingbot strips your URLs from search indexes. Keeping your AI rules gathered together under a descriptive comment helps avoid accidents during future site maintenance.

Another frequent issue is expecting noindex meta tags to work on pages disallowed in robots.txt. If a page is blocked via robots.txt, crawlers never fetch the HTML, meaning they never see the noindex meta tag inside the page head. You can confirm your syntax and catch conflicting rules by running your file through our free Robots.txt Tester.

To check live requests from suspected scrapers, review your raw server access logs. Look for spikes in hits coming from a single IP address or requests containing empty user-agent strings, and cross-reference them with the official documentation published by Google Developers to verify legitimate search crawler hostnames.

Frequently Asked Questions

How to block bots from my website?

To stop legitimate web crawlers and AI bots, add specific Disallow directives for their user-agent strings inside your robots.txt file. For aggressive scrapers and spam bots that deliberately ignore site instructions, use a Web Application Firewall like Cloudflare or your server .htaccess file to drop requests based on IP address, user-agent headers, or behavioral challenge scores.

How to block bots on Discord?

To control bots on Discord, open your Server Settings and select Roles. Ensure that the @everyone role does not have administrator privileges, webhook management access, or mention permissions. For public servers, adjust your Verification Level under Safety Setup to require a verified phone number or email, and install a verification bot like Wick or CaptchaBot that requires new members to solve a verification prompt before gaining chat roles.

How to block bots on Instagram?

To stop bot interactions on Instagram, switch your account to private if you run a personal profile. For professional and creator accounts, navigate to Settings and Privacy, select Hidden Words, and enable advanced comment filtering to automatically hide spam phrases. You can also turn on the Limits feature under How Others Can Interact With You to temporarily mute interactions from accounts that do not follow you or only recently followed you.

How to block bots on YouTube?

To suppress automated comment bots on YouTube, open YouTube Studio, click Settings in the left sidebar, and choose Community. Under the Automated Filters tab, enable Hold potentially inappropriate comments for review or select Increase strictness. You can also paste common spam phrases and URLs into the Blocked words field and check the Block links box to prevent accounts from spamming malicious URLs in your comment sections.