I don't know whether to allow or block AI crawlers
There's no universally right answer on training, but there is a wrong one: blocking the search bots by accident while trying to block training.
Decide per bot, not per company, because the three jobs have different costs. Blocking a training crawler like GPTBot or ClaudeBot keeps your content out of future models. Blocking a search crawler like OAI-SearchBot, PerplexityBot or Bingbot removes you from AI answers that cite and link you, which is direct lost traffic.
4 tools for this
Each one is free, private and runs in your browser. No sign-up, no watermark.
AI Robots.txt Generator
Generate a correct robots.txt policy for every AI crawler (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and 11 more) with per-bot allow/block toggles and a plain-language line on what each choice costs. Free, in your browser.
AI Crawler Access Checker
Check whether GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bingbot and every other AI crawler can read your site. Real RFC 9309 robots.txt matching with the deciding rule quoted, plus Cloudflare default-blocking detection. Free, instant, no signup.
AI Crawler Log Analyzer
Drop a server access log and see which AI crawlers actually visit: per-bot hit counts, first and last seen, most-fetched pages, status-code health, daily trend, and the pages no AI crawler has ever touched. Parsed in your browser: logs contain visitor IPs and never leave your device.
GigAI GEO Audit
One URL in, one AI-visibility report out: crawler access with Cloudflare detection, JS rendering, citability scored with published methodology, retrieval chunks, answer snippets and schema. Each finding linked to the free tool that fixes it.
How to fix it
Follow these steps and you'll have it sorted in a couple of minutes.
- 1
See what you block today
Run the Access Checker on your live robots.txt. Many sites are blocking crawlers from a copy-pasted template nobody chose deliberately.
- 2
Separate training from search
Training bots build future models. Search bots build the indexes AI answers cite from. User-fetch bots read a link someone shared. They are separate decisions.
- 3
Pick a policy
The common deliberate choice is to block training while keeping every search and user-fetch bot allowed, so you stay citable without feeding model training.
- 4
Generate the rules
The AI Robots.txt Generator emits a correct group per bot and shows the concrete cost of each block before you ship it.
- 5
Deploy and re-check
Put the file at your domain root, then re-run the Access Checker on the live URL to confirm a CDN hasn't rewritten it.
- Blocking Googlebot or Bingbot to 'block AI'. That removes you from ordinary search, and Bing's index also powers ChatGPT Search.
- Assuming Google-Extended is a crawler. It's a control token for Gemini training and grounding. Blocking it does not affect your Google Search rankings.
- Expecting robots.txt to undo the past. It governs future crawling only and cannot remove content from a model already trained on it.
- Forgetting subdomains. robots.txt is per-host, so a policy on your main domain does nothing for blog.example.com.
Frequently asked questions
More problems we solve
Ran into one of these too? Here's the fix.
- ChatGPT and other AI tools can't see my websiteWork through the three causes in order. Access first, rendering second, content last, because fixing content on a page crawlers can't reach changes nothing.
- I don't know whether AI crawlers are actually visiting my siteAllowing a crawler in robots.txt is permission, not proof. Only your logs show whether the crawlers came.
- My website needs a robots.txt fileControl what search engines crawl, and avoid the classic mistake of accidentally blocking your entire site.
Keep exploring
Related tools
Problems we solve
Definitions
From the blog
By file type
Common tasks
Next steps
Ready? Start with AI Robots.txt Generator
Free, private and instant: everything runs in your browser.