Robots.txt Generator Guide
Reference for drafting robots.txt rules, sitemap directives, crawler sections, and launch checks without blocking public pages.
Quick answer
Use the Robots.txt Generator to draft crawler access rules and sitemap directives for a site.
What this tool does
The tool helps assemble robots.txt sections for general crawlers, selected AI crawlers, blocked private paths, and sitemap discovery.
Supported input
- User-agent names
- Allow and Disallow path rules
- Sitemap URL
- Common private path presets
- Copy-ready robots.txt output
Data handling and processing behavior
Processing is handled in the browser for this tool based on the current public implementation. Avoid entering sensitive internal URLs unless you have reviewed the implementation and your own data handling requirements.
Step-by-step use
- Choose a general crawler section
- Add public allow rules only when needed
- Add private or account paths to disallow
- Add the production sitemap URL
- Copy the output and review it as plain text
Practical workflow
Draft robots.txt before launch, but verify it after deployment as a plain-text file. The rules should guide crawlers without blocking public hubs such as tools, docs, Blog, guides, workflows, or localized pages. Use the SEO Publishing Workflow to check robots rules beside sitemap URLs, page-level robots tags, canonicals, and hreflang entries.
Review example
For a tools site, explicitly allow public folders such as /tools, /blog, and /docs, then disallow only admin, dashboard, checkout, or draft API paths. After publishing, fetch the live robots.txt as plain text and confirm each directive appears on its own line. Then verify sitemap URLs are still reachable and indexable before launch.
Practical handoff note
For robots.txt handoff, pair every rule with its intent: block admin, allow public tools, point to sitemap, or manage crawler load. Test important URLs after generating rules, because a short disallow pattern can hide valuable pages. Keep robots directives separate from noindex and canonical decisions.
Common errors
Blocking public hubs. Check broad rules before publishing.
Using robots.txt for access control. Private data needs authentication or server controls.
Forgetting the sitemap. Sitemap discovery belongs in the robots.txt file for many sites.
Limits
Robots.txt guides crawling. It does not remove indexed content, protect secrets, or replace page-level noindex directives.
Next steps
- SEO Publishing Workflow — review robots rules with sitemap, canonical, and page metadata checks
- Meta Robots Tag Generator — prepare page-level indexing directives
- Sitemap URL Checker — review sitemap URL lists
- Canonical URL Generator — align indexable URLs