Back to SEO Tools
Robots.txt Guide
Test and debug your robots.txt rules with our free Robots.txt Tester. Validate directives, check crawl access, and generate rules — all in your browser.
Common Robots.txt Directives
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Specifies the crawler the rules apply to | User-agent: Googlebot |
| Disallow | Blocks the crawler from accessing a path | Disallow: /private/ |
| Allow | Overrides a disallow for a specific path | Allow: /public/ |
| Sitemap | Points crawlers to your XML sitemap | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Sets a delay in seconds between requests | Crawl-delay: 10 |
Best Practices
Always reference your sitemap
Add a Sitemap directive at the bottom of robots.txt to help crawlers discover all your important pages.
Test before deploying
Use our tester to validate your rules and verify which URLs are blocked before pushing to production.
Be specific with paths
Use precise paths rather than broad blocks. Never disallow entire directories unless absolutely necessary.
Respect crawl budgets
Block low-value pages (admin panels, duplicate content) to help crawlers focus on indexing your important content.