robots.txt Generator

How to use the robots.txt Generator

1. Click + Add rule group to create a block for a specific crawler.
2. Set the User-agent (use * for all crawlers, or a specific bot name like Googlebot).
3. Add Disallow or Allow rules for specific paths.
4. Optionally add a sitemap URL and crawl-delay.
5. Click Generate, then copy or download the file.

Common bot names

• * — All crawlers
• Googlebot — Google web crawler
• Bingbot — Microsoft Bing
• GPTBot — OpenAI web crawler
• ClaudeBot — Anthropic web crawler