← All tools
Robots.txt Generator & Path Tester
Generate a conservative robots.txt and test sample paths.
Robots.txt
Runs in your browser when possible — we don’t need your files on our servers.
Quick presets
Presets show a preview first — nothing is replaced until you confirm.
Group 1
Sitemaps
User-agent: * Disallow: /admin Disallow: /private Sitemap: https://example.com/sitemap.xml # Generated with Teeny Tools — robots.txt controls crawling, not security.
Path tester
Approximates RFC 9309: picks the most specific matching user-agent group, then the longest matching rule (supports * wildcards and a trailing $ end-anchor). Ties favor Allow, matching common crawler behavior.
robots.txt guides crawlers — it does not protect private data. Prefer authentication for anything sensitive.
Next: try the Website Analyzer for a broader page check.