AI Robots.txt Generator: Allow or Block Every AI Crawler, DeliberatelyNew
Fifteen AI crawler tokens, one toggle each, grouped by operator and labeled by what the bot actually does: model training, AI search indexing, or on-demand user fetching. Flip a toggle and read exactly what that block costs before you ship it. Three presets cover the sane policies: maximum AI visibility, visible-but-not-trained-on, and block-AI-keep-search. Output is a clean block to merge into your robots.txt, or a complete file.
What is the ai robots.txt generator?
The AI Robots.txt Generator builds a correct robots.txt policy for 15 AI crawler tokens with one toggle each, grouped by operator and labeled by role: model training, AI search indexing, or on-demand user fetching. Blocking a bot immediately shows what it costs, and three presets cover the sane policies. Output is a merge-ready block or a complete file, generated entirely in your browser.
Most sites' AI crawler policy was written by a template: a copy-pasted 'block AI' snippet from 2023 that predates OAI-SearchBot and Claude-SearchBot and therefore blocks AI search citations along with training, or nothing at all, which is also a policy, just not a chosen one. This generator makes the choice explicit and per-bot. Every token is listed under its operator with its role stated (GPTBot trains models. OAI-SearchBot builds the index ChatGPT Search cites from. ChatGPT-User fetches links users share: three different decisions, not one), and the moment you set a bot to Block, the tool shows the concrete cost in plain language: lost citations, dead link-reading, missing training presence. The 'no training' preset implements the most popular deliberate policy correctly: it blocks GPTBot, ClaudeBot, meta-externalagent, Bytespider, CCBot and the Google-Extended and Applebot-Extended control tokens, while keeping every search-index and user-fetch bot allowed, so you stay citable in ChatGPT Search, Claude, Perplexity, Google and Bing while opting out of model training. The 'block AI' preset deliberately keeps Googlebot and Bingbot allowed, because blocking those removes you from ordinary search: a mistake that turns a content-policy decision into a traffic catastrophe. Output is commented, spec-correct (one User-agent group per token, Allow: / or Disallow: /), and ready to paste.
- Difficulty:
- Easy
- Typical time:
- ~15s
- Processing:
- 100% browser processing
Last updated
How to use the ai robots.txt generator
- 1
Start from a preset
Pick the policy closest to your intent: full visibility, no-training, or block-AI. Presets set all 15 toggles at once, correctly.
- 2
Adjust individual bots
Flip any toggle. Blocked bots show the plain-language cost immediately, so every deviation from the preset is an informed one.
- 3
Choose block or full file
Keep the default merge block for an existing robots.txt, or enable full-file mode to add a User-agent: * group and your Sitemap URL.
- 4
Copy or download robots.txt
Deploy it at your domain root (https://example.com/robots.txt). Each subdomain needs its own file.
- 5
Verify with the Access Checker
Fetch your live file with the AI Crawler Access Checker to confirm every verdict matches your intent, and to catch a CDN rewriting your file.
What AI Robots.txt Generator includes
One toggle per bot, grouped by operator
OpenAI's three tokens, Anthropic's three, Perplexity's two, plus Google, Microsoft, Apple, Meta, ByteDance and Common Crawl. Each with its own switch, because 'AI' is not one decision.
The cost of every block, before you ship it
Set a bot to Block and the row immediately shows what you lose: ChatGPT Search citations for OAI-SearchBot, link-reading for ChatGPT-User, the Common Crawl corpus for CCBot. No silent foot-guns.
Three honest presets
Maximum AI visibility (allow all), visible-but-not-trained-on (block training tokens, keep search and user-fetch), and block-AI-keep-search (which still allows Googlebot and Bingbot, because removing yourself from search is never the intent).
Spec-correct output
One User-agent group per token with an explicit Allow: / or Disallow: /, optional explanatory comments, optional complete-file mode with a * group and Sitemap line. Nothing a parser could misread.
Pairs with the Access Checker
Generate, deploy, then verify with the AI Crawler Access Checker, which quotes your new rules back to you with full RFC 9309 matching. Check → fix → check.
Runs entirely in your browser
The policy you configure is generated locally and never uploaded. Share links encode your toggle state in the URL fragment, which never reaches a server.
Why use our ai robots.txt generator
Turn a default into a decision
Whatever your stance on AI training, it should be yours. Per-bot toggles with stated costs replace the 2023 template someone else wrote with a policy you actually chose.
Stop blocking your own citations
The most damaging pattern in the wild is blocking OAI-SearchBot or Bingbot while trying to block training. The role labels and preset logic make that mistake hard to commit.
Explain the policy to anyone
The generated comments document what each group does, so the next developer (or lawyer) reading your robots.txt understands the policy without archaeology.
Merge-friendly output
By default you get just the AI block, ready to append to your existing file without touching your current rules. Full-file mode exists when you're starting fresh.
Built for the way you work
From quick one-off fixes to daily workflows, see how people put this tool to use.
- Site owner
Opt out of training, stay citable
The no-training preset is the policy most owners actually want: models can't train on your content, but ChatGPT Search, Claude, Perplexity and Bing can still cite and link you.
- Publisher
Implement an editorial AI policy precisely
When the organization decides its stance on AI use of content, this turns the memo into correct directives: with documentation comments legal can read.
- Developer
Stop hand-writing bot groups
Fifteen user-agent groups typed by hand is fifteen chances for a typo that silently does nothing. Generate them, paste once, done.
- Agency
A defensible default for every client
Ship each client a deliberate AI policy with the trade-offs documented in the file itself, instead of whatever their theme's robots.txt happened to contain.
What this tool does not do
Boundaries stated plainly, with the right tool for each neighbouring job.
- It doesn't build general crawl rules (path disallows, crawl-delay, custom groups): that's the Robots.txt Generator. Robots.txt Generator does that.
- It can't enforce anything against bots that ignore robots.txt. Enforcement lives in your firewall or CDN bot management.
Supported formats
Accepts Toggles, and produces robots.txt and TXT, all processed locally in your browser.
- Toggles
- robots.txt
- TXT
Frequently asked questions
Recommended tools
AI Crawler Access Checker
Check whether GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bingbot and every other AI crawler can read your site. Real RFC 9309 robots.txt matching with the deciding rule quoted, plus Cloudflare default-blocking detection. Free, instant, no signup.
Robots.txt Generator
Build a valid robots.txt with user-agent groups, allow/disallow rules, crawl-delay, sitemap and host directives, with live output, presets and lint warnings. 100% in your browser.
AI Crawler Log Analyzer
Drop a server access log and see which AI crawlers actually visit: per-bot hit counts, first and last seen, most-fetched pages, status-code health, daily trend, and the pages no AI crawler has ever touched. Parsed in your browser: logs contain visitor IPs and never leave your device.
GigAI GEO Audit
One URL in, one AI-visibility report out: crawler access with Cloudflare detection, JS rendering, citability scored with published methodology, retrieval chunks, answer snippets and schema. Each finding linked to the free tool that fixes it.
llms.txt Generator
Generate a spec-shaped llms.txt from your sitemap or URL list, with an honest answer to the question every other generator dodges: Google has said its AI search doesn't use llms.txt. Here's who it actually helps, and a good file in ten seconds if you want one.
AI Schema Checker
Audit a page's JSON-LD against the seven schema types AI retrieval leans on, verify each one's load-bearing properties, and get every gap routed to the GigAI generator that fills it. A checker, not another generator.
Common problems, solved
Hit a snag? Here are quick fixes for the issues people run into most.
I deployed the file but AI bots still fetch my site.
Confirm the live file at yourdomain.com/robots.txt matches what you generated (CDNs and plugins can rewrite it), give crawlers up to 24 hours to re-read it, and remember Bytespider has a history of ignoring robots.txt entirely: blocking it reliably requires a firewall rule.
I blocked AI training but my content still shows up in AI answers.
Two different mechanisms. AI answers that cite you come from search indexes and live fetching (OAI-SearchBot, PerplexityBot, Bingbot…), which the no-training preset deliberately allows. And robots.txt can't remove content from models that already trained on it. It only governs future crawling.
Should the AI block go before or after my existing rules?
Order doesn't matter to compliant parsers. Each crawler obeys the most specific group naming it, wherever it sits in the file. Put the block wherever reads best. Just don't duplicate a token in two groups with contradictory rules.
Cloudflare shows different robots.txt content than my file.
Cloudflare's managed robots.txt feature injects its own section above yours. Disable it in the dashboard if you want your generated policy to be the whole policy: otherwise Cloudflare's Disallows apply for the tokens it lists.
Get the most out of it
Google-Extended and Applebot-Extended are control tokens, not crawlers: they govern how Googlebot's and Applebot's crawls may be used for AI. Blocking them doesn't reduce crawl traffic, and doesn't affect your search rankings.
Never block Googlebot or Bingbot to 'block AI'. That removes you from ordinary search. Bing's index also powers ChatGPT Search, so a Bingbot block silently costs you ChatGPT citations too.
robots.txt is public. Your AI policy is readable by anyone at /robots.txt. Which is fine, and also why a well-commented file is worth generating.
Re-visit the policy quarterly: new tokens appear (OAI-SearchBot and Claude-SearchBot didn't exist when most 'block AI' templates were written), and this generator's list is kept current.
Blocking a bot doesn't un-train existing models. The decision affects future crawls only: make it for the future, not as a retraction.
What's new
Recent updates and improvements to the ai robots.txt generator.
Initial release: 15 AI tokens with per-bot toggles and block-cost explanations, three presets (visibility / no-training / block-AI-keep-search), merge-block and complete-file output modes, commented spec-correct generation, shareable configuration links.
Keep exploring
Related tools
Problems we solve
Definitions
From the blog
Explore categories
By file type
Common tasks
Your privacy is built in
The generator runs entirely in your browser: toggle state and output never leave your device, and share links keep the configuration in the URL fragment, which browsers don't send to servers. There is nothing to log and nobody logging it.
- Runs in your browser
- No uploads
- Nothing stored
Ready to try the ai robots.txt generator?
Free, private and instant. AI Robots.txt Generator runs right in your browser.