Build a production-ready robots.txt with allow/disallow rules, crawl-delay, sitemap directives and per-bot blocks. Validated as you type.
Decide which pages you want crawlers to check on your website.
The tool takes seconds to generate the Robots.txt file.
No paywall, no sign-up. Generate and download as many robots.txt files as you need.
Entrepreneurs who run e-commerce businesses and have sensitive pages such as checkout pages that need not be crawled by search engine crawlers can use this tool.
Website owners who want to control the visibility on different search engines including the search engine such as Google.
If you write bulk content, this tool can generate the robots.txt file so you can control the visibility of your content.
A simple markdown file at the root of your site — yourdomain.com/robots.txt — that tells Google, Bing and other search engines what your site is about and which pages to read, surface and cite.
User-agent: * Allow: / Disallow: /admin/ Disallow: /checkout/ Disallow: /*?*sessionid= User-agent: GPTBot Allow: / Sitemap: https://acme.com/sitemap.xml
A four-step flow from your website to the search engines answer a user sees.
Website owners who want to control the visibility on different search engines..
ChatGPT, Claude, Perplexity & Gemini read it before parsing your full site.
The H2 sections become a curated index of what matters on your site.
When someone asks a search engine a related question, your pages are surfaced & cited.
Creating a robots.txt file is surprisingly simple. All you need is a plain text file named robots.txt placed in the root directory of your website.
Open a text editor like Notepad or VS Code.
Add your crawl directives.
Save the file as robots.txt.
Upload it to yourdomain.com/robots.txt.
Example:
User-agent: *
Disallow:
Sitemap: https://yourdomain.com/sitemap.xmlIf you're not comfortable writing code, you can use the Mohit SEO Training Robots.txt Generator to instantly generate a valid robots.txt file without worrying about syntax errors.
A robots.txt file works using a few simple commands called directives.
Directive | What it does |
|---|---|
| Specifies which crawler the rule applies to |
| Prevents crawling of a page or folder |
| Explicitly allows crawling of a blocked path |
| Provides the location of your XML sitemap |
Example:
User-agent: *
Disallow: /private/
Allow: /private/public.pdf
Sitemap: https://example.com/sitemap.xmlFor most websites, these four directives are all you'll ever need.
Not exactly.
A robots.txt file can block crawlers from accessing your pages, but it does not guarantee removal from search results.
For example, to block Bing:
User-agent: Bingbot
Disallow: /This prevents Bingbot from crawling your website. However, if other websites link to your pages, they may still appear in Bing's search results.
Important: If your goal is to remove pages from Google, use a noindex meta tag rather than robots.txt.
A well-structured robots.txt helps search engines spend their crawl budget on your valuable pages instead of unnecessary URLs.
Keep the file clean and easy to read.
Block admin and login areas.
Include your XML sitemap.
Avoid blocking CSS or JavaScript files.
Test the file after making changes.
Use comments to document complex rules.
A minimal robots.txt is often more effective than an overly complicated one.
Use the Disallow directive followed by the folder path.
Example:
User-agent: *
Disallow: /wp-admin/This blocks crawlers from accessing everything inside the wp-admin directory.
You can block multiple folders as well:
User-agent: *
Disallow: /private/
Disallow: /backup/
Disallow: /temp/Remember that the path is always relative to your website's root.
The easiest method is through your browser.
Simply visit:
https://yourdomain.com/robots.txt
If the file exists, you'll immediately see its contents.
You can also verify it using:
Google Search Console
SEO crawling tools
Browser developer tools
Checking your robots.txt regularly ensures that important sections of your website haven't been blocked accidentally.
A robots.txt file does not directly increase rankings or traffic.
Its real purpose is improving how search engines crawl your website.
Better crawl efficiency
Faster discovery of important pages
Reduced crawling of duplicate or low-value URLs
A single incorrect rule like:
User-agent: *
Disallow: /can unintentionally block your entire website from being crawled, leading to serious indexing issues.
Always review your robots.txt before publishing it.
Add your sitemap using the Sitemap directive.
Example:
User-agent: *
Disallow:
Sitemap: https://yourdomain.com/sitemap.xmlIf your website has multiple sitemaps, simply list each one:
Sitemap: https://example.com/post-sitemap.xml
Sitemap: https://example.com/page-sitemap.xmlIncluding your sitemap helps search engines discover your newest pages more quickly.
You can create different rules for different crawlers.
Example:
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /This allows Googlebot to crawl the website while asking all other compliant bots not to.
Keep in mind that robots.txt is a voluntary standard—malicious bots may ignore these instructions entirely.
No.
This is one of the biggest misconceptions in technical SEO.
Robots.txt only controls crawling, not indexing.
If you need to remove a page from Google, use one of these methods instead:
Add a noindex meta tag
Return a 404 or 410 status code
Use Google Search Console's URL Removal Tool for temporary removal
Blocking the page in robots.txt alone can actually prevent Google from seeing the noindex instruction, making removal more difficult.