Robots.txt Tester

The rule that decides is the longest one, not the first one

Almost every explanation of robots.txt you will read says the first matching rule wins. It does not. Google — and, since RFC 9309, everyone claiming to follow the standard — compares the length of the matching path patterns and applies the longest. A Disallow: / at the top of a group does not stop an Allow: /blog/ five lines down from winning for /blog/post, because /blog/ is six characters and / is one.

When two matching patterns are the same length, Allow wins. That is the whole tie-break. It is why Allow: /folder and Disallow: /folder in the same group means the folder is crawlable.

Only one group applies — this is the one that costs money

A crawler reads the whole file, picks the single group whose user-agent token best matches its own name, and then behaves as though the rest of the file does not exist. It does not merge your Googlebot group with your * group.

So the moment you add three lines of Googlebot-specific rules to relax one path, every restriction in your * group stops applying to Googlebot. The faceted-navigation blocks, the internal search block, the print-view block — all of it, gone, for the only crawler that matters. This is the single most common serious robots.txt bug, and it is invisible until your crawl stats go sideways. Set the crawler above to Googlebot on the sample file and watch which group is used.

The wildcards

• Every pattern is a prefix match. Disallow: /admin blocks /admin, /administrator and /admin-panel. If you meant the folder, write /admin/.
• * matches any run of characters, including none and including slashes. /*? blocks every URL with a query string.
• $ anchors the end, and only at the end of the pattern. /*.pdf$ blocks /a/b.pdf but not /a/b.pdf?download=1. A $ anywhere else is just a dollar sign.
• Disallow: with nothing after it means allow everything. It is not a block. Disallow: / is the block, and the missing slash is a one-character difference between "crawl my whole site" and "crawl none of it".

Blocking a page is not removing it

A disallowed URL can still be indexed. Google will not fetch it, but if enough pages link to it, it will appear in results as a bare URL with no title and no description, because Google knows it exists and is not allowed to look. To keep something out of the index you have to let it be crawled and serve noindex. Block it in robots.txt instead and Google never fetches the page, never sees the noindex, and the page stays indexed indefinitely. The two directives cancel each other out, and the one that loses is the one you wanted.

What this tool does not know

It tests the file you paste against the matching rules as published. It cannot tell you whether that file is the one actually being served, whether your CDN is returning a 404 for /robots.txt on some edges, or whether a crawler claiming to be Googlebot really is. A robots.txt that returns a 5xx is treated by Google as "disallow everything" for up to about thirty days — no rule in it matters at that point.