Robots.txt Validator
Paste your robots.txt content to check it for syntax errors, unrecognized directives, and common mistakes before publishing it.
Recommended Settings
Pro Tips
- •Every Disallow or Allow directive must appear after a User-agent line - rules listed before any User-agent are ignored by crawlers
- •This checks syntax and structure, not whether your rules will produce the crawling behavior you actually intend - review the logic of your rules separately
- •Paths in Disallow and Allow directives should generally start with a forward slash, since they're matched against the URL path
- •This validator checks the file's structure locally - it doesn't fetch or check your live robots.txt file from a URL
Most Popular
Most users paste their robots.txt content here right before publishing it to catch typos or structural mistakes
When to Use This Tool
Check your robots.txt file for mistakes before uploading it to your live site.
Investigate whether a syntax error in your robots.txt might be causing unexpected crawler behavior.
Understand the correct structure and common mistakes in the robots.txt format.
Validate a robots.txt file you inherited from a previous developer or site migration.
How It Works
Parse each line, skipping blank lines and comments
Check that each directive is a recognized robots.txt directive with a properly formatted value
Verify that Allow and Disallow rules always follow a User-agent line, and flag empty user-agent blocks
Report all errors and warnings found, along with the line number for each
100% Private
Files never leave your device. All processing happens locally in your browser.
Lightning Fast
Powered by Client-side syntax validation against the Robots Exclusion Protocol for optimal performance on modern browsers.
Open Source
Built with verified, open-source libraries. Fully transparent.
Frequently Asked Questions
Does this check my live robots.txt file automatically?
No. This tool validates whatever text you paste into it locally. To check a live file, copy its contents from yourdomain.com/robots.txt and paste them here.
What's the difference between an error and a warning?
Errors indicate something structurally wrong, like an unrecognized directive or a rule appearing in an invalid position, which crawlers may not interpret correctly. Warnings flag likely mistakes, like a path not starting with a slash, that might still technically work but deviate from common convention.
Is my data sent to a server?
No. All validation happens locally in your browser using JavaScript. Nothing you enter is transmitted anywhere.
Does passing validation guarantee my crawling rules work as intended?
No. This tool checks syntax and structure, not whether your specific rules achieve your intended crawling behavior. Review your Allow and Disallow paths carefully to make sure the logic matches what you actually want blocked or permitted.
Can I validate multiple User-agent blocks at once?
Yes. Paste your entire robots.txt file, including all User-agent blocks, and the validator will check each one in sequence.