How to Create a Robots.txt File for Your Website

Learning how to create a robots.txt file for your website comes down to placing a simple, plain text file in your root directory to instruct search engines where they can and cannot crawl. You can build one manually using any text editor, generate it automatically through a browser tool, or manage it directly inside your content management system. Getting this setup right prevents bots from wasting server bandwidth on private paths while keeping your public content open for indexing.

Which robots.txt creation method should you choose?

Every crawler looks for your directives at a single, standard URL: example.com/robots.txt. Because the file is just plain text encoded in UTF-8, you have multiple ways to create and publish it. The best method depends on how your site is hosted and how much technical control you want.

If you run a standard custom website or need clean directives fast without syntax errors, you can use the free Robots.txt Generator. If you prefer working directly in code or use an established CMS, manual editing or built-in plugin controls work just as well.

What should a robots.txt file contain?

A functional robots.txt uses simple key-value directives. Crawlers read the file from top to bottom, matching against rules that apply to their designated user-agent name.

Here are the core components every standard file uses:

• User-agent: Names the specific crawler the following rules apply to, such as Googlebot, Bingbot, or an asterisk (*) for all crawlers.
• Disallow: Specifies URL paths that the specified crawler must not fetch. An empty disallow line means nothing is blocked.
• Allow: Explicitly permits crawling of a specific subfolder or file that sits inside an otherwise disallowed directory.
• Sitemap: Provides the absolute URL to your XML sitemap, pointing crawlers to your indexable pages.

A minimal file that permits all crawlers and points to an XML sitemap looks like this:

User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml

If you do not have your XML sitemap ready yet, you can create one with a Sitemap.xml Generator before adding the sitemap directive.

Method 1: Using an online robots.txt generator

Using a web generator is the fastest way to assemble directives without risking formatting mistakes. Typographic errors like missing slashes or incorrect casing can cause crawlers to misunderstand your rules entirely.

To generate your file:

1. Open the Links to Wild Robots.txt Generator, which builds a valid robots.txt file with per-bot rules, common disallows, and the sitemap directive while running entirely in your browser.
2. Select your default crawling rule for all search engines (allow everything or block everything).
3. Check the boxes for common directories you want to keep bots out of, such as administration dashboards, shopping cart checkouts, or internal search result pages.
4. Specify dedicated rules for individual crawlers if you want separate access policies for search crawlers versus AI training bots.
5. Enter your absolute XML sitemap URL into the sitemap field.
6. Copy the generated text or download the output file named robots.txt.
7. Upload the file to the root directory (often named public_html or www) of your web host via SFTP or your hosting file manager.

Method 2: Creating a robots.txt file manually in a text editor

If you work with static sites or prefer direct file management, you can create the file locally in seconds with any plain text editor.

1. Open a plain text editor such as Notepad, TextEdit, or VS Code. Do not use word processors like Microsoft Word, as they insert hidden formatting characters that break parsing.
2. Set the file encoding to UTF-8. Robots.txt must follow valid UTF-8 formatting according to standards outlined on Google Developers.
3. Add your user-agent group. To address every crawler, type User-agent: * on the first line.
4. Add your disallow paths. For example, if you want to block an internal scripts directory, add Disallow: /scripts/. Every path must start with a forward slash.
5. Add your XML sitemap location at the bottom using its full URL: Sitemap: https://example.com/sitemap.xml.
6. Save the file with the exact name robots.txt. Keep all letters lowercase. A file named Robots.txt or robots.TXT will not be recognised by web crawlers.
7. Connect to your server using SFTP, SSH, or your web hosting control panel, and upload the file to your root web directory.

Method 3: How to create a robots.txt file for a WordPress website

WordPress handles crawling differently out of the box. By default, it generates a virtual robots.txt file dynamically using PHP. If no physical file exists in your root folder, WordPress serves this virtual file automatically.

You have two ways to edit or create your file in WordPress:

1. Check your search engine visibility settings: In your WordPress dashboard, navigate to Settings > Reading. Ensure the checkbox labelled Discourage search engines from indexing this site is unchecked. Checking that box adds a universal disallow rule (Disallow: /) to your virtual file, blocking your entire site.
2. Use an SEO plugin: Most standard SEO plugins for WordPress include a file editor tool. In your plugin's tools or dashboard settings, look for the robots.txt editor. You can paste your custom directives directly into the text field and save, which updates the virtual file without requiring server access.
3. Upload a physical file: If you upload a real robots.txt file to your root directory via FTP or your hosting file manager, server web software (like Nginx or Apache) serves that static file instead of the WordPress virtual file. This overrides any plugin-generated virtual directives.

Common mistakes and how to test your file

A single typo in your robots file can accidentally de-index sections of your site or prevent critical stylesheets from rendering. Watch for these common issues:

• Blocking CSS and JavaScript files: Modern search engines render pages like a browser to evaluate content and layout. If you disallow /assets/ or /js/, search crawlers cannot render your pages correctly, which can harm rankings.
• Using Disallow to hide pages from search results: Disallowing a page in robots.txt does not remove it from Google search results. It only prevents bots from crawling the content. If external links point to that URL, Google can still index the address without snippet text. To completely remove a page from search results, allow crawling and use a meta tag built with a Noindex Tag Generator instead.
• Incorrect path casing: Robots.txt paths are case-sensitive. The directive Disallow: /private/ will not block /Private/ or /PRIVATE/.
• Missing forward slashes: Directives like Disallow: admin match any URL containing "admin" anywhere in the path, rather than just the /admin/ folder. Always start directory paths with a forward slash.

Once you publish your file, verify that it resolves correctly in your browser by visiting https://example.com/robots.txt. You should see plain text without HTML wrapper tags. You can then test specific URL rules against crawlers using the Robots.txt Tester to confirm which line matches your priority pages.

Frequently Asked Questions

A page I blocked is still showing in Google. How do I get it out?

Unblock it. While the URL is disallowed Google cannot fetch the page, so it cannot see the noindex that would remove it; it keeps the URL in the index with no description. Remove the Disallow line, serve <meta name="robots" content="noindex">, and wait for a recrawl. Use the Removals tool in Search Console if you need it gone this week.

Do I need a robots.txt at all?

Not if you want everything crawled. A missing file means exactly that, and a 404 on /robots.txt is perfectly normal. What you must avoid is the file returning a 5xx error: Google treats that as "unknown" and will back off crawling the whole site until it resolves. A broken robots.txt is worse than no robots.txt.

Should I block the AI crawlers?

Decide per bot, because they do different things. Google-Extended governs Gemini and AI Overviews and has no effect on ordinary Search ranking. GPTBot is training. ChatGPT-User is live retrieval; blocking it means your pages are never fetched when someone asks ChatGPT a question your site answers. For most sites that last one is a referral channel, not a threat.

Does Crawl-delay work?

Bing and Yandex honour it. Google ignores it entirely and always has. If Googlebot is genuinely overloading your server, the crawl-rate setting in Search Console is the lever, and a 503 with a Retry-After header is the emergency one.

Does my robots.txt cover subdomains?

No. Every host gets its own, so blog.example.com and shop.example.com each need their own file at their own root. This is the usual reason a staging subdomain gets indexed: the block lives on the main domain, where nothing needed blocking.