Robots.txt Generator
Generate a robots.txt file to control how search engine crawlers access your website. Add rules for specific user agents, disallow paths, and link your sitemap.
Recommended Settings
Pro Tips
- •robots.txt must be placed at the root of your domain (like example.com/robots.txt) to be recognized by crawlers
- •User-agent: * applies the rule to all crawlers - use a specific bot name (like Googlebot) to target rules at just one crawler
- •Disallow blocks crawling of a path, but doesn't guarantee a page won't appear in search results if it's linked from elsewhere - use a noindex meta tag for that instead
- •Not all crawlers respect Crawl-delay, and Google specifically ignores it in favor of settings in Google Search Console
Most Popular
Most sites start with a single User-agent: * block disallowing admin or private paths, plus a sitemap reference
When to Use This Tool
Create a basic robots.txt file as part of launching a new website.
Prevent search engines from crawling admin panels, staging areas, or duplicate content paths.
Point crawlers to your sitemap.xml file so they can discover your pages more efficiently.
Create different crawling rules for different search engines or bots.
How It Works
For each user-agent block, list the crawler name along with its disallowed and allowed paths
Combine all blocks together, separated by blank lines, following the standard robots.txt format
Append a Sitemap directive pointing to your sitemap file, if provided
100% Private
Files never leave your device. All processing happens locally in your browser.
Lightning Fast
Powered by Client-side text generation following the Robots Exclusion Protocol for optimal performance on modern browsers.
Open Source
Built with verified, open-source libraries. Fully transparent.
Frequently Asked Questions
Where do I put my robots.txt file?
It must be placed at the root of your domain, accessible at yourdomain.com/robots.txt. Placing it anywhere else means crawlers won't find or respect it.
Does Disallow prevent a page from appearing in search results?
Not necessarily. Disallow only blocks crawling - if a disallowed page is linked from elsewhere, search engines may still index the URL itself without visiting it. Use a noindex meta tag on the page itself to reliably prevent indexing.
Is robots.txt legally enforceable?
No. It's a voluntary standard that well-behaved crawlers (like Googlebot) respect, but nothing technically prevents a bot from ignoring it. It shouldn't be relied upon to protect genuinely sensitive or private data.
Is my data sent to a server?
No. The file is generated entirely locally in your browser using JavaScript. Nothing you enter is transmitted anywhere.
Should every website have a robots.txt file?
It's not strictly required, but it's considered good practice, especially for pointing crawlers to your sitemap or preventing crawling of low-value pages like admin panels or duplicate content.