robots.txt Generator
Create a robots.txt file with custom bot access rules, AI crawler controls, sitemap references, and crawl delay settings.
🤖 Configuration
Default Access
Path Rules
Bot Access Control
✓ = allowed to crawl · unchecked = blocked
Generated robots.txt
# robots.txt generated by RPDI Free Tools
# https://rpdi.us/tools/robots-txt-generator
User-agent: *
Allow: /
Disallow: /admin
Disallow: /private
# Blocked bots
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: YandexBot
Disallow: /
📋 How to Use
- Configure your access rules and bot permissions above
- Copy or download the generated
robots.txtfile - Upload it to your website's root directory
- Validate it using Google Search Console's robots.txt Tester
Need help with technical SEO?
Get a Free SEO ConsultationFrequently Asked Questions
What is a robots.txt file and do I need one?
robots.txt is a text file at the root of your website that tells search engine crawlers which pages they can and cannot access. While not mandatory, it's considered an SEO best practice. It helps you prevent search engines from indexing duplicate content, admin pages, or sensitive directories.
Should I block AI crawlers like GPTBot in robots.txt?
It depends on your strategy. Blocking AI crawlers (GPTBot, Claude, CCBot) prevents your content from being used to train AI models, but it also prevents your site from appearing in AI-generated answers. Our generator includes all major AI crawler options so you can decide per-bot.
Can robots.txt prevent my pages from appearing in Google?
robots.txt can prevent Google from crawling a page, but if other sites link to it, Google may still index the URL (showing it with no description). To fully prevent indexing, use a "noindex" meta tag instead. robots.txt is best for managing crawl budget.