What Does Robots.txt Contain? Key Directives Explained

If you are wondering what does robots.txt contain, it is a simple plain-text file made of crawler directives that tell search engine bots which parts of your site they can or cannot request. At minimum, it normally contains a User-agent line specifying which bot the rule applies to, one or more Disallow or Allow paths, and an optional link pointing to your XML sitemap. It sits in your website's top-level root directory and functions purely as a traffic-management guide for automated bots, rather than a security fence or a guarantee against indexing.

How robots.txt works on a web server

Every search crawler checks for a file named robots.txt in your domain root before fetching any other page. For example, if Googlebot wants to crawl https://example.com/blog/guide, it first issues an HTTP request to https://example.com/robots.txt. If the file exists and is accessible, the bot parses the text from top to bottom, matches its own identifier against the user-agent groups, and obeys the first matching rule it finds.

If you build your file manually or use a tool like the Links to Wild Robots.txt Generator, the file must be accessible via HTTP or HTTPS at the root path. Placing the file in a subfolder like example.com/assets/robots.txt does nothing; crawlers will never look there. The server must also return a text/plain content type with UTF-8 encoding so international crawlers can read the characters without parsing errors.

Robots.txt operates under the Robots Exclusion Protocol, an internet standard formalized by the Internet Engineering Task Force. Crawlers follow these rules on an honour system. Legitimate crawlers run by Google, Bing, and major search engines read and respect these rules consistently, while malicious scrapers and spam bots will disregard them completely.

The core lines a robots.txt file normally contains

A functional robots.txt file breaks down into distinct instruction blocks. Each block addresses one or more crawlers and lists the URL paths that bot is permitted or forbidden to fetch.

Directive	Purpose	Example Syntax
User-agent	Identifies which bot or automated crawler the rules apply to.	User-agent: Googlebot
Disallow	Tells the bot which URL path or directory it must not fetch.	Disallow: /checkout/
Allow	Explicitly permits access to a specific subfolder or file inside a disallowed path.	Allow: /checkout/thank-you
Sitemap	Declares the absolute URL of an XML sitemap to help engines discover URLs.	Sitemap: https://example.com/sitemap.xml

User-agent

The User-agent directive tells the crawler who the following lines are written for. You can address all standard crawlers at once using an asterisk wildcard (User-agent: *), or you can address specific crawlers such as Googlebot, Bingbot, or GPTBot. When a bot reads your file, it looks for a group specifically matching its own name. If it finds one, it obeys only that specific group and ignores the generic wildcard group entirely.

Disallow

The Disallow directive tells matching user-agents not to download the specified URL prefix. A forward slash followed by a path tells the crawler to stay away from anything starting with that pattern. For example, Disallow: /admin/ stops bots from fetching example.com/admin/ and example.com/admin/login. An empty disallow value (Disallow:) tells bots that all sections of the site are open for crawling.

Allow

The Allow directive creates exceptions inside a blocked path. When search engines evaluate rules, the most specific path length wins. If you disallow a whole media directory but need bots to access the images inside it so they can render your layout, you can add an explicit allow rule:

Disallow: /wp-content/
Allow: /wp-content/uploads/

Sitemap

Unlike user-agent, allow, and disallow rules, the Sitemap directive is independent of bot groupings. It can appear anywhere in the file, though standard convention puts it at the very top or bottom. It requires a full, absolute URL (including https://) rather than a relative path, giving bots an immediate pointer to your sitemap index.

Pattern matching with wildcards

Most modern search engine crawlers support basic pattern matching characters inside robots.txt paths:

• Asterisk (*): Matches any sequence of characters, including empty strings. Writing Disallow: /*?sort= prevents bots from requesting faceted navigation URLs containing that parameter.
• Dollar sign ($): Matches the end of a URL. Writing Disallow: /*.pdf$ prevents crawlers from fetching files that end specifically with the .pdf extension, while leaving a URL like /pdf-converter/ open.

Order matters less than specificity in Google's parser. If an Allow directive and a Disallow directive match the same URL, Google prioritizes the directive with the longer, more specific character length. When you want to see how these conflicting rules evaluate, you can run them through the Links to Wild Robots.txt Tester to confirm which rule takes precedence.

A complete robots.txt example

Here is an example showing how these directives come together on a standard publishing or business website:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /search/
Disallow: /*?*filter=

User-agent: Googlebot-Image
Disallow: /private-gallery/

Sitemap: https://example.com/sitemap.xml

In this example, general search bots are barred from back-end administrative paths, customer checkout flows, internal search queries, and filtered parameter strings. Google's image crawler is singled out to keep private gallery photos out of Google Images, and the full sitemap location is declared at the end.

When robots.txt matters and when it does not

Understanding what robots.txt does not do is just as important as knowing what it contains. A frequent misconception is that robots.txt is a security protocol or an indexation block.

Robots.txt controls crawling, not indexing

Blocking a URL in robots.txt tells Google not to fetch the content of that page. It does not stop Google from listing that URL in search results. If other websites, or other internal pages on your own site, link to that blocked URL, Google still knows the page exists. Because it cannot crawl the page to see what is on it, Google will index the bare URL, often displaying the link without a snippet or description.

If you want a page out of the search index entirely, do not block it in robots.txt. Instead, ensure search engines can crawl it, and serve a noindex robots directive on the page itself. You can generate clean code for this using the Links to Wild Noindex Tag Generator. If you block the page in robots.txt at the same time, the crawler cannot see your noindex tag, and the page stays in the index.

Robots.txt is not a security measure

A robots.txt file is publicly readable. Anyone can type /robots.txt into their browser and see the exact directories you are trying to hide. Listing staging directories, administrative login panels, or confidential documents in your disallow rules merely advertises their locations to bad actors. Keep sensitive files secured behind server authentication, passwords, or IP restrictions rather than robots.txt rules.

When robots.txt is genuinely useful

Robots.txt is practical for managing crawl budget and server stability. It is best used for:

• Faceted navigation: Preventing crawl traps caused by millions of filter and sorting parameter combinations in e-commerce stores.
• Search results: Blocking internal search query pages (like /search?q=) to keep low-value auto-generated URLs out of the crawl queue.
• Server load: Keeping resource-heavy scripts or endless dynamic archives from draining your server bandwidth.
• AI crawler permissions: Defining how generative engines and training scrapers treat your site using an AI crawler checker to review your coverage.

Generating and maintaining your file

You can create a robots.txt file by writing it in a plain text editor like Notepad or VS Code and uploading it to your web server root via FTP or your hosting control panel. Alternatively, CMS platforms like WordPress, Shopify, or Webflow often generate a default file for you.

If you prefer a clean manual setup without syntax mistakes, you can use the Links to Wild Robots.txt Generator to build a valid file with per-bot rules, common disallows, and your sitemap directive already formatted. Because it runs directly in your browser, you can copy the output straight into your server root or CMS settings.

Official crawler documentation from Google Developers specifies that robots.txt files must not exceed 512 kibibytes. Keep your rules concise, test them whenever site structures change, and confirm your file returns a healthy 200 HTTP status code.

Frequently Asked Questions

Is robots.txt necessary?

No, a robots.txt file is not strictly necessary if you want everything on your site crawled. When a robots.txt file is missing, the server returns a 404 status code, which search engines interpret as permission to crawl the entire domain. However, you must avoid the file returning a 5xx server error. Google treats server errors as an unknown state and backs off crawling the entire site until the error resolves. A broken robots.txt file is far worse than having no file at all.

A page I blocked is still showing in Google. How do I get it out?

You need to unblock it first. While the URL is disallowed in robots.txt, Google cannot fetch the page, which means it cannot see the noindex directive that would remove it. Because external or internal links still point to it, Google keeps the URL in search results without a description. Remove the Disallow line, serve <meta name="robots" content="noindex"> on the page, and wait for Google to recrawl it. You can use the URL Removals tool in Google Search Console if you need it dropped immediately.

Should I block AI crawlers in robots.txt?

You should decide per bot rather than applying a blanket ban, because different AI crawlers serve distinct purposes. Google-Extended controls access for Gemini and AI Overviews, but has no effect on ordinary Google Search indexing. GPTBot collects data for model training. ChatGPT-User handles real-time web retrieval during live user queries; blocking it means your site will not be fetched when a user asks a question your content answers. For most websites, real-time citation bots represent a discovery channel rather than a problem.

Does the Crawl-delay directive work?

The Crawl-delay directive works on Bing and Yandex, but Google ignores it completely and always has. If Googlebot is putting excessive load on your server, adjust the crawl rate setting within your Google Search Console account. If your server is under immediate strain, returning a 503 HTTP status code with a Retry-After header is the standard way to ask crawlers to pause temporarily.

Does my robots.txt cover subdomains?

No, robots.txt rules apply only to the exact protocol and host where the file lives. A file at example.com/robots.txt does not control blog.example.com or staging.example.com. Each subdomain requires its own separate file at its own root directory. This distinction is the most common reason staging sites get indexed by mistake: developers put blocking directives on the main domain root, leaving the staging subdomain completely open.