How to Block GPTBot Using Robots.txt and Firewalls

If you need to know how to block GPTBot, add two directives to your site's robots.txt file: specify User-agent: GPTBot on one line, followed immediately by Disallow: / on the next. This instruction tells OpenAI's automated crawler that it cannot access any page across your domain for training foundation models. The change takes effect as soon as your server serves the updated file and OpenAI fetches your current instructions.

Deciding Between Robots.txt and Network Blocks

There are two primary methods for restricting AI scrapers: protocol directives and server-level firewall rules. Each serves a different technical purpose.

The standard method relies on your robots.txt file. This file sits at the root of your domain (for example, example.com/robots.txt) and outlines crawling permissions for automated bots. OpenAI officially documents that its primary training crawler respects standard robots exclusion rules. If your goal is simply to prevent your content from being scraped for future model training while keeping your server configuration clean, this is the recommended approach.

If you want to know how to block permanently at the infrastructure level regardless of whether a crawler checks your permissions, you will need a Web Application Firewall (WAF) or web server configuration block. A firewall inspects inbound requests and drops connections from OpenAI's published IP ranges or user-agent strings before the bot can load a single byte of your pages. This approach requires more maintenance, as IP ranges can change over time.

Before changing files on your server, you can audit your current configuration using the Free AI Crawler Checker. It inspects your directives directly in your browser to show which crawlers currently have access.

Auditing Your Directives with an AI Crawler Checker

Checking your setup before and after making edits ensures you have not introduced syntax issues that leave doors open to scrapers. The AI Crawler Checker evaluates how bots read your site without sending any data to external servers.

1. Navigate to your live site and copy the text at example.com/robots.txt.
2. Open the AI Crawler Checker tool page.
3. Paste your raw text into the input field.
4. Review the generated report to see the verdict for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended alongside the specific rule governing each bot.

Because the parsing and matching run entirely in your browser, the page has no backend and makes no network request with the text you enter. If you need to build fresh directives from scratch for multiple systems, you can also generate clean syntax with our AI Crawler Control Generator.

Editing Your Robots.txt File in a Text Editor

The simplest direct way to apply changes is by modifying your physical robots.txt file on your hosting server.

1. Connect to your web hosting server using SFTP, SSH, or your hosting control panel file manager.
2. Locate your web root directory (often named public_html, www, or htdocs).
3. Open the existing robots.txt file in a plain text editor like VS Code, Notepad, or a terminal editor such as Nano. If the file does not exist, create a new plain text file named robots.txt in that directory.
4. Add the dedicated record for the scraper: User-agent: GPTBot Disallow: /
5. Save the file and upload it back to your server root.
6. Open a browser tab and visit https://example.com/robots.txt to confirm the new lines render accurately in plain text.

If you want to allow indexing across the rest of your site while targeting only specific directories for protection, you can adjust the paths. For example, Disallow: /private/ prevents the bot from scraping that folder while leaving the rest of the site open.

Updating Robots Directives in WordPress

If your website runs on WordPress, the robots.txt file is often generated dynamically by the core software or by an SEO plugin rather than existing as a static file on disk.

1. Log in to your WordPress administrative dashboard (usually at example.com/wp-admin).
2. Navigate to your installed SEO plugin's tools or file editor section. In most common SEO plugins, this setting sits under general settings, tools, or advanced technical controls.
3. Look for an option labeled "Edit robots.txt" or "Robots File Editor".
4. Insert your rules at the bottom of the editor window: User-agent: GPTBot Disallow: /
5. Click save or update changes.
6. Verify that your site serves the new lines by opening example.com/robots.txt in an incognito window.

If you do not use an SEO plugin that manages robots directives, you can generate a custom static file using our free Robots.txt Generator and upload it directly to your WordPress root directory via FTP. A physical file on the server always overrides WordPress dynamic output.

Blocking Traffic at the CDN or Firewall Level

If you want to know how to block permanently without depending on crawlers respecting voluntary standards, configure a firewall rule at your content delivery network (CDN) or hosting edge.

1. Log in to your CDN or edge security provider dashboard (such as Cloudflare).
2. Navigate to the Security or WAF (Web Application Firewall) section, then select Custom Rules.
3. Create a new firewall rule and give it a descriptive name like "Block OpenAI Scrapers".
4. Under the matching criteria, configure the rule to inspect the incoming User-Agent string. Set the field to User Agent, select the "contains" operator, and enter GPTBot as the value.
5. Set the action to Block or Managed Challenge.
6. Deploy the rule. Inbound HTTP requests carrying that user-agent string will now receive a 403 Forbidden status code or be dropped at the network edge before hitting your server.

Common Robots.txt Errors and How to Verify the Result

Even small formatting mistakes can cause crawlers to ignore your instructions. Below are the most frequent pitfalls to watch out for when writing your rules.

Incorrect Directive Order

Directives apply downward within a block. A Disallow rule must always sit below the corresponding User-agent line. If you write Disallow: / above User-agent: GPTBot, the parser attaches that disallow rule to whichever bot group sat above it, leaving GPTBot unrestricted.

Typographical Errors

Bot user agents are specific tokens. If you write User-agent: GPT-Bot or User-agent: GPTbot instead of GPTBot, the crawler will not match the record. It will treat your site as having no specific rules for it and fall back to the default User-agent: * directive.

Confusing Crawl Blocking with Search Removal

A rule in robots.txt stops the fetch; it does not set an indexing command. A page blocked from crawling can still theoretically show up in search results if other domains link to it. Conversely, if you want search engines to drop a page from an index entirely, you need a noindex meta tag or HTTP header. If you disallow a page in robots.txt, the crawler never fetches the page to see the noindex directive in the HTML. Use our Noindex Tag Generator to create the correct tags when your objective is index removal rather than crawl management.

Testing Your Production File

Once you deploy your changes, paste your published content into the AI Crawler Checker again. Confirm that GPTBot displays a red blocked status and that the rule shown next to the verdict matches your intended disallow path.

Frequently Asked Questions

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot is OpenAI's training crawler used to collect datasets for future language models. What fetches your page in real time during a user conversation is ChatGPT-User, and what indexes your site for ChatGPT search features is OAI-SearchBot. Blocking GPTBot alone leaves both of those active, meaning users can still search for and reference your content inside ChatGPT.

How do I block permanently if bots ignore robots.txt?

To block permanently and enforce the restriction technically, apply firewall rules at your hosting server or CDN. You can block requests containing the bot's user-agent token or drop connections coming from OpenAI's published CIDR IP blocks. This returns an HTTP 403 Forbidden code or terminates the connection before the scraper reads your content.

Why does an audit tool say a bot is allowed when my file lists it?

This usually happens for one of three reasons: the bot token is spelled differently from the published name, the disallow directive sits above the User-agent line instead of below it, or the crawler has its own record and the disallow rule you intended for it was placed inside the general wildcard group (User-agent: *).

Is a blocked AI crawler the same as a noindex tag?

No. robots.txt stops the crawler from making a fetch request to read the page. A noindex tag instructs a search engine not to display the URL in its index. A crawler must actually fetch the HTML to see a noindex tag; blocking the page via robots.txt prevents the crawler from discovering that tag.