Enter your domain or paste your robots.txt, and all 15 AI crawler tokens get a verdict: allowed, blocked, or unspecified. Each one is computed the way crawlers actually compute it, following RFC 9309 for group specificity, wildcards, longest-match precedence and Allow beating Disallow on ties, and each verdict quotes the exact line that decided it. It also detects Cloudflare, which now blocks AI crawlers by default regardless of what your file says.
The AI Crawler Access Checker reports whether GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Bingbot and 10 more AI crawler tokens may fetch your site, computed with real RFC 9309 matching: user-agent group specificity, wildcard paths, longest-match precedence and Allow beating Disallow on ties. Each verdict quotes the exact robots.txt line that decided it. It also detects Cloudflare, which blocks AI crawlers by default on new zones regardless of robots.txt.
Most robots.txt checkers get the matching wrong: they treat the file as a list instead of user-agent groups, ignore wildcard patterns, and miss that Allow beats Disallow on a specificity tie. This checker implements the Robots Exclusion Protocol properly: consecutive User-agent lines share a group, duplicate groups merge, the most specific agent match wins, path rules support * and $, and the longest matching rule decides. It then applies that to every AI crawler token across OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google (Googlebot, Google-Extended), Microsoft (Bingbot, which is the index ChatGPT Search retrieves through), Apple (Applebot-Extended), Meta (meta-externalagent), ByteDance (Bytespider) and Common Crawl (CCBot). Every verdict quotes the matching rule and its line number, explains what the bot does, and states plainly what blocking it costs you. And because robots.txt stopped being the whole story in 2025: when you fetch by URL, the response headers are inspected for Cloudflare, whose 'AI bots' firewall setting now blocks AI crawlers by default on new zones, and whose managed robots.txt feature injects Disallow rules above your own. Thousands of sites are blocked without anyone having edited anything. This is the free tool that tells you if you're one of them.
Last updated
Type example.com and the checker fetches https://example.com/robots.txt automatically. Or paste the file's contents to analyze a draft or staging copy privately.
Each of the 15 tokens shows Allowed, Blocked or Allowed (unspecified), grouped by operator, with the deciding rule quoted and the bot's role explained.
If the site is behind Cloudflare, the checker says so from response evidence and tells you which dashboard toggles (Security → Bots → AI bots. Managed robots.txt) can override your file.
Every blocked verdict links to the AI Robots.txt Generator, which builds a correct per-bot policy with plain-language explanations of each choice.
Paste the new file or re-fetch the URL. The share link reproduces the report so a teammate can verify the same evidence.
OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, ByteDance and Common Crawl. Every token that decides your AI visibility, grouped by operator with its role labeled: training, search index, or on-demand user fetch.
RFC 9309 as crawlers implement it: user-agent groups merged by token, exact match beating prefix beating *, wildcard paths (* and $), longest-match precedence, Allow winning ties. Most online checkers get at least one of these wrong.
No verdict without evidence: each bot's result shows the exact robots.txt line (with line number) that decided it, or states that no rule names the bot and crawling defaults to allowed.
Cloudflare blocks AI crawlers by default on new zones and can inject a managed robots.txt above yours. Fetch by URL and the response headers plus in-file markers (Content-Signal lines, managed-section comments) are checked, with the exact dashboard setting named.
Blocking OAI-SearchBot is not like blocking GPTBot. Every blocked verdict explains the concrete consequence: lost ChatGPT Search citations, dead link-reading, missing training presence, so you decide with the trade-off in view.
Pasted files never leave your browser. URL fetches go through a stateless proxy that fetches, returns and stores nothing. Either way there's no account and no log of what you checked.
Agencies, CDN defaults, WordPress plugins and copy-pasted robots.txt templates block AI crawlers silently. The most common finding this tool produces is a Disallow someone shipped without deciding to.
A file that looks permissive can block through group precedence, and one that looks blocking can allow through a longer Allow rule. The spec-correct matcher shows the truth, not the appearance.
Blocking training bots while allowing search and user-fetch bots is a coherent, popular policy. The per-role labels let you check your file implements the policy you chose, not one a template chose for you.
Curious how competitors or publishers handle AI crawlers? Enter any domain: the checker fetches its public robots.txt and shows its policy with the same rigor.
From quick one-off fixes to daily workflows, see how people put this tool to use.
The first question of AI-era SEO. In ten seconds you know whether the answer is a robots.txt rule, a Cloudflare default, or neither, before you spend a cent on visibility tooling.
Crawl access used to mean Googlebot. Now an audit that doesn't check OAI-SearchBot and PerplexityBot misses the fastest-growing referral channel. One URL per client, evidence included.
If your organization decided to block AI training, verify the file implements it: correct tokens, correct rules, no wildcard-group loopholes, and that the search bots you still want are not caught in the blast radius.
Group precedence and longest-match rules produce surprising outcomes. Paste the file, see which group each bot actually obeys and which rule wins for the path.
Boundaries stated plainly, with the right tool for each neighbouring job.
Accepts URL, robots.txt and TXT, and produces Report and Share link, all processed locally in your browser.
See how the ai crawler access checker fits into a step-by-step journey with related tools.
Generate a correct robots.txt policy for every AI crawler (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and 11 more) with per-bot allow/block toggles and a plain-language line on what each choice costs. Free, in your browser.
Drop a server access log and see which AI crawlers actually visit: per-bot hit counts, first and last seen, most-fetched pages, status-code health, daily trend, and the pages no AI crawler has ever touched. Parsed in your browser: logs contain visitor IPs and never leave your device.
One URL in, one AI-visibility report out: crawler access with Cloudflare detection, JS rendering, citability scored with published methodology, retrieval chunks, answer snippets and schema. Each finding linked to the free tool that fixes it.
Paste your robots.txt to parse it into User-agent groups and catch the mistakes that block Google: a stray Disallow: /, rules before any User-agent, unknown directives and a missing sitemap. 100% in your browser.
Generate a spec-shaped llms.txt from your sitemap or URL list, with an honest answer to the question every other generator dodges: Google has said its AI search doesn't use llms.txt. Here's who it actually helps, and a good file in ten seconds if you want one.
Audit a page's JSON-LD against the seven schema types AI retrieval leans on, verify each one's load-bearing properties, and get every gap routed to the GigAI generator that fills it. A checker, not another generator.
geo-toolsA reference for the fifteen AI crawler tokens that decide your visibility in ChatGPT, Claude, Perplexity and Google, what each one actually does, and what blocking it costs you.
geo-toolsCloudflare now blocks AI crawlers at the firewall for new zones, before robots.txt is ever read. Here is why a permissive robots.txt proves nothing, and exactly where to look.
Hit a snag? Here are quick fixes for the issues people run into most.
robots.txt permission is necessary, not sufficient. Check the Cloudflare warning (firewall blocking happens before robots.txt is read), then confirm with real evidence: drop your server access log into the AI Crawler Log Analyzer to see who actually arrives.
That's the point of the flag. Cloudflare's 'AI bots' setting (Security → Bots) blocks AI crawlers at the network edge regardless of robots.txt, and it's enabled by default for zones created since mid-2025. Only the dashboard shows its state: the flag tells you to go look.
No rule names it and no wildcard group restricts it, so crawling defaults to allowed under the protocol. It's fine. Adding an explicit Allow group only helps as documentation of intent.
Bytespider has a documented history of ignoring robots.txt. The file is a request that well-behaved crawlers honor. Enforcement against a misbehaving one needs a firewall rule or your CDN's bot management.
Blocking Google-Extended does NOT remove you from Google Search or AI Overviews: it's a usage-control token for Gemini training, not a crawler. Blocking Googlebot removes you from everything. Almost never do that.
ChatGPT Search retrieves through Bing's index: blocking Bingbot silently removes you from ChatGPT Search results even with every OpenAI bot allowed.
The popular 'block AI' copy-paste templates predate OAI-SearchBot and Claude-SearchBot: sites using them often block training AND search citations without realizing the second part.
Check the file at your domain root, not a copy: CDNs and plugins can rewrite robots.txt at serve time, and Cloudflare's managed feature injects rules your repo never contained.
robots.txt is per-subdomain: blog.example.com has its own file, and a policy on www does nothing for it.
Recent updates and improvements to the ai crawler access checker.
Initial release: RFC 9309 matcher (group merging, specificity, wildcards, Allow-wins-ties), 15 AI crawler tokens across 9 operators with per-role block-cost explanations, Cloudflare header + managed-robots.txt + Content-Signal detection, URL fetch with automatic /robots.txt resolution, shareable reports.
Pasted robots.txt files are parsed entirely in your browser and never uploaded. URL mode fetches the public file through our stateless proxy. It fetches, returns and stores nothing, and the checked URLs are not logged. Check drafts and staging files freely.
Free and instant, AI Crawler Access Checker processes your file securely and removes it right after.
