Meta Robots Generator
Tick the directives you want and get the meta tag and the matching X-Robots-Tag header, with the combinations that cancel each other out flagged as you go.
Nothing you type here is uploaded, stored or sent anywhere. It all runs in your browser.
noindex and robots.txt are not two ways of doing the same thing
They are two different mechanisms and using both on one URL breaks both. Disallow stops Google fetching the page. noindex is on the page. If Google cannot fetch the page, it cannot read the tag, so the noindex is never seen and never acted on — and the URL can still get indexed from inbound links, showing in results as a title-less, description-less entry that you now have no way to remove.
The sequence that works is: allow the page to be crawled, serve noindex, wait for Google to recrawl and drop it, and only then — if you even still want to — block it in robots.txt. Doing it in the other order is one of the most common ways a page gets stuck in the index for months.
A bot-specific tag replaces the generic one, it does not add to it
If a page carries both <meta name="robots" content="noindex"> and <meta name="googlebot" content="max-snippet:50">, Googlebot reads the second one and ignores the first entirely. It is not noindexed for Google. This is the same trap as robots.txt groups, and it has the same cause: people assume the specific block is an override of one setting rather than a replacement of all of them. If you write a googlebot tag, repeat every directive you still want in it.
The meta tag and the header do the same thing in different places
- Use the meta tag for HTML. It must be in the
<head>. A robots tag that ends up in the<body>, which happens when something earlier in the head is malformed, is ignored. - Use the header for everything else. A PDF, a CSV, an image, a JSON endpoint — none of them have a head to put a tag in.
X-Robots-Tagis the only way to noindex them, and it is why old PDFs are the most commonly forgotten indexed content on a site. - They combine, restrictively. If a page has both, the most restrictive directive wins. You cannot relax a header with a tag.
- Injecting the tag with JavaScript works, but late. Google will see it on the render pass. Every other crawler, and Google's first pass, will not.
noindex, follow expires
noindex, follow was the standard recipe for paginated archives and faceted pages for years: keep the page out of the index, let the link equity flow through. Google has since said plainly that this does not hold up over time — a page that stays noindexed gets crawled less and less, and eventually the links on it stop being followed at all. It still behaves as follow in the short term. Do not build an architecture that depends on it working forever.
Questions people ask
Do I need to add index, follow to my pages?
No. Those are the defaults for every page that has no robots tag at all. Adding <meta name="robots" content="index, follow"> is harmless and does absolutely nothing. If a CMS is adding it to every page, that is a tell that the plugin was written to look busy.
What is the difference between nofollow here and rel="nofollow" on a link?
Scope. rel="nofollow" applies to the one link it is on. nofollow in a robots tag applies to every link on the page at once, including your own navigation. People reach for the page-level version to deal with one bad outbound link and quietly cut their internal linking with it.
Does noindex remove a page from Google immediately?
No. It takes effect the next time Google crawls the page, which could be days or weeks depending on how often that URL gets crawled. If you need it gone now, use the Removals tool in Search Console for the temporary block and let the noindex do the permanent work underneath it.
Does Bing use the same directives?
Mostly. noindex, nofollow, noarchive and nosnippet are understood by Bing. The max-snippet, max-image-preview and max-video-preview family is a Google extension, as is unavailable_after. Bing also has nocache, which is its own thing.
Why does this tool not generate a google-extended meta tag?
Because there is no such thing. Google-Extended is a robots.txt user-agent token that controls whether your content is used for Gemini and Vertex AI training. It has no meta-tag equivalent, and a <meta name="google-extended"> tag is ignored by everything. Tick the box and this tool will show you the two robots.txt lines that actually do it.
Related tools
Deciding what not to index is half the job
The other half is making the pages you do index worth finding. That is content and links, in that order.