How to Check Robots.txt and Test URL Blocking Rules

The fastest way to learn how to check robots txt on any website is to append /robots.txt directly to the end of the root domain in your browser address bar. For instance, typing https://example.com/robots.txt will load the plain-text file immediately if one exists. If you see raw text containing directives like User-agent: and Disallow:, the file is live and readable by web crawlers.

A robots.txt file gives instructions to automated bots about which parts of a site they should or should not crawl. While looking at the text file confirms its existence, it does not tell you whether a complex rule actually blocks a specific URL for a particular search engine. Testing the syntax against real crawler behaviours requires a dedicated check.

Which Method Should You Use to Check Robots.txt?

Different verification methods serve different purposes depending on whether you own the site, are troubleshooting a specific URL, or are auditing an entire catalog:

• Direct browser check: Best for a 5-second check to see if a robots.txt file exists and what basic rules are published.
• Browser-based rule tester: Best for checking whether a specific URL is allowed or blocked for Googlebot, Bingbot, or AI bots without needing account access.
• Google Search Console: Best for verifying which version of the file Google currently has cached and requesting a recrawl of an updated file.
• CMS backends (WordPress, Shopify): Best when you need to inspect or modify the generated file directly inside your publishing platform.
• Desktop crawlers (such as Screaming Frog): Best for auditing thousands of URLs at once to see which pages are blocked across an entire site.

Method 1: Test Rules and Crawler Logic with an In-Browser Tester

Checking the raw text in your browser can be deceptive. A rule like Disallow: /admin does not behave the same way across all bots, and path precedence rules mean that a longer Allow: directive can override a broader Disallow:. You can use the free Links to Wild Robots.txt Tester to see exactly which rule decides access for any specific URL.

1. Open the target website's /robots.txt file in your browser, select all the text, and copy it.
2. Go to the Robots.txt Tester.
3. Paste the raw text into the file input area.
4. Enter the specific URL path you want to verify (for example, https://example.com/category/product-1 or /category/product-1).
5. Select your target crawler from the list, such as Googlebot, Bingbot, or specific AI scrapers.
6. Run the check to see whether the URL is allowed or blocked, along with the exact line number of the matching rule that decided the outcome.

The parser and matcher run locally inside your browser, meaning no network requests are sent with your pasted rules. If you need to make changes or create a clean file from scratch after testing, you can build valid directives with a robots.txt generator.

Method 2: Check Googlebot Status via Google Search Console

If you own or manage the site, Google Search Console provides the definitive answer on how Googlebot reads your file. Google maintains a dedicated Robots.txt Report within Search Console that displays the current live version Google has fetched, the timestamp of the last fetch, and any syntax warnings.

1. Log in to Google Search Console and select your domain property.
2. In the left-hand navigation menu, scroll down and click on Settings.
3. Under the "Crawling" section, locate the robots.txt row.
4. Click to open the report. You will see whether Google encountered any fetch errors, the date and time of the last fetch attempt, and the complete text content Google has on record.
5. To test an individual page, use the URL Inspection tool in the top search bar. Paste any URL from your domain, run the live test, and expand the "Crawl" section to check the "Crawl allowed?" field. If blocked by robots.txt, Search Console will say so explicitly.

Method 3: How to Check Robots.txt in WordPress

WordPress generates a virtual robots.txt file dynamically by default if no physical file exists on the server root. To check and manage this file, you can inspect it in your browser or through standard SEO plugins.

1. Navigate to yoursite.com/robots.txt in your browser to view the dynamic output. By default, WordPress often blocks /wp-admin/ while allowing /wp-admin/admin-ajax.php.
2. Log in to your WordPress dashboard. First, ensure the site is not globally blocked: navigate to Settings > Reading and verify that "Search engine visibility" is unchecked. If checked, WordPress outputs a rule blocking everything: Disallow: /.
3. If you use an SEO plugin (such as Yoast SEO or Rank Math), navigate to the plugin's Tools menu and select the File Editor.
4. The editor will display your active robots.txt file. If changes are needed, you can edit the text directly inside WordPress without needing FTP access.

Method 4: How to Check Robots.txt in Shopify

Shopify automatically generates a standard robots.txt file for every store. It blocks system paths like checkout pages, internal search queries, cart pages, and admin routes out of the box.

1. Visit yourstore.com/robots.txt to view the live file.
2. Log in to your Shopify admin panel.
3. Go to Online Store > Themes.
4. Click the three dots (...) next to your active theme and select Edit code.
5. Look in the Templates folder for a file named robots.txt.liquid.
6. If this template exists, your store has customized rules. If it does not exist, Shopify is serving its default rules. You can add standard rules using Shopify's Liquid filters or inspect the raw output directly.

Method 5: Bulk-Check Robots.txt Blocks in Screaming Frog

When working on large sites, checking individual paths by hand is impractical. Desktop crawlers like Screaming Frog SEO Spider allow you to check the crawl status of thousands of URLs against your robots.txt file simultaneously.

1. Open Screaming Frog and enter your root domain in the top bar.
2. Go to Configuration > Robots.txt > Settings.
3. Select your preferred crawl mode: choose Respect robots.txt to make the spider obey your rules, or Ignore robots.txt if you want to crawl blocked URLs and report their status.
4. Run the crawl. Once completed, navigate to the Internal tab.
5. Look at the Status column. Any URL blocked by a rule will be flagged as "Blocked by robots.txt".
6. Switch to the Robots.txt tab in the bottom window to see the exact line of code that triggered the block for the selected URL.

Common Robots.txt Mistakes and How to Avoid Them

A single typo in a robots.txt file can de-index key sections of a site or fail to block sensitive paths. Watch for these common implementation errors when checking your file:

1. Path Case-Sensitivity Issues

Robots.txt directive names (like Disallow or User-agent) are case-insensitive, but file paths are case-sensitive. If your rule is Disallow: /Admin/, a bot is still permitted to crawl /admin/. Always inspect the exact case used in your actual URL structures.

2. Using Robots.txt Instead of Noindex

A disallow rule tells search engines not to crawl a page. It does not guarantee the page will not be indexed. If external websites link to a blocked URL, search engines can still index the URL without crawling the body content, resulting in an empty snippet in search results. If you need a page completely removed from search results, allow it to be crawled and implement a proper meta tag using a noindex generator instead.

3. Subdomain and Protocol Separation

A robots.txt file applies strictly to the exact protocol and host it is hosted on. For example, https://example.com/robots.txt has zero authority over https://subdomain.example.com/ or an insecure http:// version. Staging sites hosted on subdomains require their own dedicated robots.txt file to keep bots out.

4. Rule Precedence Confusion

Search engines like Google evaluate rules based on length, not order. The longest matching path wins. Consider this block:

Even though the Disallow directive appears first, /products/featured is allowed for Googlebot because the Allow directive has a longer matching character count (18 characters versus 10 characters).

5. User-Agent Group Isolation

Standard crawlers match only the single most specific user-agent group that applies to them; they do not combine groups. If your file includes a dedicated group for User-agent: Googlebot and a separate wildcard group for User-agent: *, Googlebot reads only the Googlebot group and ignores the wildcard block completely.

If you discover technical crawl issues or link distribution problems that are holding back organic growth across complex sections of your site, reviewing our pricing for comprehensive link acquisition campaigns can help build the external equity needed to support competitive sections once crawling is clean.

Frequently Asked Questions

How do I check if my robots.txt is working or not?

To check if your robots.txt file is working, inspect the HTTP response header in your browser network tab or run the file through an in-browser matcher. The URL must return an HTTP 200 OK status code and serve raw plain text (text/plain). You can verify individual path behaviour directly with the Robots.txt Tester to confirm whether your intended rules match specific URLs.

Why does one robots.txt tool say allowed when another says blocked?

Discrepancies between testing tools usually come down to two parsing rules: group selection and path precedence. Many basic tools improperly merge crawler-specific groups with wildcard groups, whereas actual search engines only follow the most specific matching group. Additionally, some tools mistakenly follow the first rule listed rather than prioritizing the longest matching path.

How do I find the robots.txt file of any website?

To find the robots.txt file on any website, navigate to the domain root and add /robots.txt to the end of the URL. For example, enter https://example.com/robots.txt into your browser. If the server returns a 404 error, a blank page, or redirects to the homepage, the website does not have an active robots.txt file, which means all crawlers are allowed unrestricted access.

Should I block AI crawlers in my robots.txt?

Blocking AI crawlers like GPTBot, ClaudeBot, or PerplexityBot is a business and content protection decision, not an SEO requirement. Restricting these bots does not affect your standard Google organic rankings. If you wish to manage AI access without hurting search performance, you can use specialized directives or review your setup with an AI crawler checker.