Free Robots.txt Generator

Q: Should I block the AI crawlers?

Decide per bot, because they do different things. Google-Extended governs Gemini and AI Overviews and has no effect on ordinary Search ranking. GPTBot is training. ChatGPT-User is live retrieval — blocking it means your pages are never fetched when someone asks ChatGPT a question your site answers. For most sites that last one is a referral channel, not a threat.

Q: Does my robots.txt cover subdomains?

No. Every host gets its own, so blog.example.com and shop.example.com each need their own file at their own root. This is the usual reason a staging subdomain gets indexed: the block lives on the main domain, where nothing needed blocking.

robots.txt

Nothing you type here is uploaded, stored or sent anywhere. It all runs in your browser.

What robots.txt does and does not do

It is not a security control. The file is public. Listing /admin tells everyone where your admin is.
Disallow is not noindex. A blocked page can still be indexed from external links — it just gets indexed with no description. To keep something out of the index, let it be crawled and serve noindex.
Never block CSS or JS. Google renders pages; blocking assets makes your site look broken to it.
It lives at the root only. /robots.txt, nowhere else, one per subdomain.

Questions people ask

A page I blocked is still showing in Google. How do I get it out?

Unblock it. While the URL is disallowed Google cannot fetch the page, so it cannot see the noindex that would remove it — it keeps the URL in the index with no description. Remove the Disallow line, serve <meta name="robots" content="noindex">, and wait for a recrawl. Use the Removals tool in Search Console if you need it gone this week.

Do I need a robots.txt at all?

Not if you want everything crawled — a missing file means exactly that, and a 404 on /robots.txt is perfectly normal. What you must avoid is the file returning a 5xx error: Google treats that as "unknown" and will back off crawling the whole site until it resolves. A broken robots.txt is worse than no robots.txt.

Should I block the AI crawlers?

Decide per bot, because they do different things. Google-Extended governs Gemini and AI Overviews and has no effect on ordinary Search ranking. GPTBot is training. ChatGPT-User is live retrieval — blocking it means your pages are never fetched when someone asks ChatGPT a question your site answers. For most sites that last one is a referral channel, not a threat.

Does Crawl-delay work?

Bing and Yandex honour it. Google ignores it entirely and always has. If Googlebot is genuinely overloading your server, the crawl-rate setting in Search Console is the lever, and a 503 with a Retry-After header is the emergency one.

Does my robots.txt cover subdomains?

No. Every host gets its own, so blog.example.com and shop.example.com each need their own file at their own root. This is the usual reason a staging subdomain gets indexed: the block lives on the main domain, where nothing needed blocking.

Robots.txt Generator