Robots.txt Generator and Tester
Allow and block rules for search and AI bots, tested rule by rule.
16 bots·URL test·No sign-up
robots.txt
URL test
| Bot | Result | Matching rule |
|---|
- GPTBot and ClaudeBot
- Google-Extended
- Sitemap line
- robots.txt download
What robots.txt is, and why AI bots matter.
robots.txt is a plain text file at the root of a site that tells crawlers which addresses they may crawl. It is not a security measure but a permission list: well-behaved bots (Googlebot, Bingbot and the bots of the large AI companies) read it and follow it.
With AI one question became two: should your content go into model training, and should it appear as a source in AI search? You can close training bots such as GPTBot, ClaudeBot and Google-Extended while leaving search bots such as OAI-SearchBot and PerplexityBot open. The llms.txt Generator also checks live which bots can reach your site.
Which bot comes for what.
AI companies’ bots do three jobs. Training bots collect content for model training; search bots build the index for AI search; user-request bots come when a person opens an address inside a chat.
Google-Extended and Applebot-Extended are not separate bots but permission tokens: Googlebot and Applebot still crawl the site; these lines only decide whether the content is used for AI training. Google Search and AI Overviews visibility depend on Googlebot.
| Bot | Company | Role |
|---|---|---|
OAI-SearchBot | OpenAI | AI search |
Claude-SearchBot | Anthropic | AI search |
PerplexityBot | Perplexity | AI search |
ChatGPT-User | OpenAI | user request |
Claude-User | Anthropic | user request |
Perplexity-User | Perplexity | user request |
GPTBot | OpenAI | training |
ClaudeBot | Anthropic | training |
Google-Extended | training | |
Applebot-Extended | Apple | training |
CCBot | Common Crawl | training |
meta-externalagent | Meta | training |
Bytespider | ByteDance | training |
Six robots.txt mistakes to avoid.
- 01
One line that closes the site
A “User-agent: * / Disallow: /” line left over from development drops the site out of search. It is the first check after launch.
- 02
Blocking CSS and JS files
Google renders a page like a browser; with style and script files closed the page is judged incomplete.
- 03
Hiding private pages with robots.txt
robots.txt is public and does not remove pages from the index; a page that must stay out needs noindex, a private page needs a password.
- 04
Confusing training with search
Closing every AI bot also closes the chance of being cited in ChatGPT and Perplexity search.
- 05
Letter case and the trailing slash
Paths are case-sensitive; “/Blog” and “/blog” differ, and a “/blog” rule also covers “/blog-post”.
- 06
Not declaring the sitemap
The Sitemap line lets every bot find the sitemap; it is also added to Search Console separately.
More free tools.
- SEO CheckerSEO + GEO Readiness Score, 303 rules
- Google Rank CheckerRank and address, top 100
- Keyword AnalysisVolume, difficulty, ideas and package
- AI Visibility CheckerYour brand in ChatGPT, Gemini and Claude answers
- Schema CheckerJSON-LD gaps, per type
- llms.txt GeneratorAI bots and an llms.txt draft
- DA PA CheckerDA, PA and Spam Score
- SERP Preview ToolTitle and description, by pixel width
- Meta Tag GeneratorTitle, description, Open Graph and X
- Schema GeneratorFAQ, organisation, local business, article, breadcrumb
- AI Bot Log AnalyzerGPTBot, ClaudeBot and others in your server log
- Word CounterWords, characters, reading time, readability
- Keyword Density CheckerOne, two and three-word phrases
- UTM BuilderCampaign links, GA4-ready
- KDV CalculatorKDV inclusive, exclusive, base and withholding
- Discount CalculatorPercent off, rate, reverse and chained discounts
- Percentage CalculatorPercent of, what percent, change and raise
- SEO ROI CalculatorA return-on-investment scenario with your own figures
- ROAS CalculatorROAS, CPA, CPC, CTR and conversion rate
- WhatsApp Link Generatorwa.me link with a message, and a QR code
The bots are in. Next, getting cited.
Opening the door to AI bots is the first step; to be cited in answers, content, structured data and authority have to work together. GEO work is part of every one of our SEO packages.
- Technical SEO audit
- AI access and llms.txt
- GEO in every package
Robots.txt, what people ask.
The most asked questions about blocking AI bots, testing and where the file lives.
Back to the ToolTo the root of the site, so it opens at yourdomain.com/robots.txt. A robots.txt in a subfolder is ignored; each subdomain has its own file.
Write “Disallow: /” under a “User-agent: GPTBot” line. This closes only OpenAI’s training bot; ChatGPT search uses OAI-SearchBot, which is managed separately.
Blocking training bots (GPTBot, ClaudeBot, Google-Extended) does not affect Google rankings. Blocking AI search bots, however, lowers the chance of being cited in ChatGPT, Claude and Perplexity answers.
No. A blocked address that gets links from other sites can stay in the index without a description. To remove a page from the index it must stay crawlable and carry a noindex tag.
Google applies the longest, most specific matching rule; at equal length Allow wins. The URL test in the tool uses the same method and shows the deciding rule for every bot.
No. Generating and testing run entirely in your browser; the content is not sent to our server.
Need help?
Reach out, we are here 24/7.
For your questions, project or ideas, our expert team is with you 24/7. The relevant team reviews your request and gets back to you.