Every AI Crawler User-Agent, and Whether to Allow It
A reference for the fifteen AI crawler tokens that decide your visibility in ChatGPT, Claude, Perplexity and Google, what each one actually does, and what blocking it costs you.
The free technical layer under AI visibility: check whether AI crawlers can reach your site, whether they can read it without JavaScript, and whether there's anything on it a model could actually cite.
11 tools available
One URL in, one AI-visibility report out: crawler access with Cloudflare detection, JS rendering, citability scored with published methodology, retrieval chunks, answer snippets and schema. Each finding linked to the free tool that fixes it.
Check whether GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bingbot and every other AI crawler can read your site. Real RFC 9309 robots.txt matching with the deciding rule quoted, plus Cloudflare default-blocking detection. Free, instant, no signup.
Generate a correct robots.txt policy for every AI crawler (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and 11 more) with per-bot allow/block toggles and a plain-language line on what each choice costs. Free, in your browser.
Drop a server access log and see which AI crawlers actually visit: per-bot hit counts, first and last seen, most-fetched pages, status-code health, daily trend, and the pages no AI crawler has ever touched. Parsed in your browser: logs contain visitor IPs and never leave your device.
See your page the way AI crawlers see it: without JavaScript. Fetch the raw HTML, count the words that survive, detect client-rendering framework markers, and diff against the rendered DOM to list exactly which content is invisible to GPTBot, ClaudeBot and PerplexityBot.
Score any page or draft on the factors a published study (KDD 2024) measured as increasing AI citation: fact density, source citations, quotability, answer-first openings, question headings and self-containment. Every sub-score shows its formula and what it counted.
See your page the way a RAG system does: split into retrieval chunks and read one at a time, with no surrounding context. Flags every fragment that collapses alone: pronoun openers, 'as mentioned above', sections that never name their subject.
Check whether every section of your page opens with a self-contained answer in 40 words or fewer. Flags throat-clearing intros, buried answers and openings that depend on missing context, with a rewrite target for each failure.
Paste your description as it appears on your site, GitHub, LinkedIn, Product Hunt, Crunchbase and G2, and see where your own profiles contradict each other: name-spelling drift, conflicting numbers, conflicting years, profiles too thin to corroborate anything.
Generate a spec-shaped llms.txt from your sitemap or URL list, with an honest answer to the question every other generator dodges: Google has said its AI search doesn't use llms.txt. Here's who it actually helps, and a good file in ten seconds if you want one.
Audit a page's JSON-LD against the seven schema types AI retrieval leans on, verify each one's load-bearing properties, and get every gap routed to the GigAI generator that fills it. A checker, not another generator.
Start with the GEO Audit: one URL, one report, every finding linked to the tool that fixes it. Or go straight to the specific check you need. No account, no usage cap, and every score shows its arithmetic.
Browse the 11 tools grouped by what they do.
Check whether GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bingbot and every other AI crawler can read your site. Real RFC 9309 robots.txt matching with the deciding rule quoted, plus Cloudflare default-blocking detection. Free, instant, no signup.
Generate a correct robots.txt policy for every AI crawler (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and 11 more) with per-bot allow/block toggles and a plain-language line on what each choice costs. Free, in your browser.
Drop a server access log and see which AI crawlers actually visit: per-bot hit counts, first and last seen, most-fetched pages, status-code health, daily trend, and the pages no AI crawler has ever touched. Parsed in your browser: logs contain visitor IPs and never leave your device.
Score any page or draft on the factors a published study (KDD 2024) measured as increasing AI citation: fact density, source citations, quotability, answer-first openings, question headings and self-containment. Every sub-score shows its formula and what it counted.
See your page the way a RAG system does: split into retrieval chunks and read one at a time, with no surrounding context. Flags every fragment that collapses alone: pronoun openers, 'as mentioned above', sections that never name their subject.
Check whether every section of your page opens with a self-contained answer in 40 words or fewer. Flags throat-clearing intros, buried answers and openings that depend on missing context, with a rewrite target for each failure.
Paste your description as it appears on your site, GitHub, LinkedIn, Product Hunt, Crunchbase and G2, and see where your own profiles contradict each other: name-spelling drift, conflicting numbers, conflicting years, profiles too thin to corroborate anything.
Generate a spec-shaped llms.txt from your sitemap or URL list, with an honest answer to the question every other generator dodges: Google has said its AI search doesn't use llms.txt. Here's who it actually helps, and a good file in ten seconds if you want one.
Audit a page's JSON-LD against the seven schema types AI retrieval leans on, verify each one's load-bearing properties, and get every gap routed to the GigAI generator that fills it. A checker, not another generator.
The mechanics behind every tool here, stated plainly, including where the evidence is thin.
OpenAI alone runs three: GPTBot trains models, OAI-SearchBot builds the index ChatGPT Search cites from, and ChatGPT-User fetches links people share. Blocking all three because you object to training silently removes you from AI answers and their referral traffic. Decide per bot, not per company.
Cloudflare now blocks AI crawlers at the firewall by default on new zones, and a managed robots.txt feature can inject Disallow rules above your own. Both act before your file is ever read, so a permissive robots.txt can coexist with a total AI block. Check Security → Bots in the dashboard.
Googlebot renders pages. GPTBot, ClaudeBot and PerplexityBot read the raw HTML and move on. A client-rendered site can rank perfectly in Google while being an effectively blank page to every AI system. Server-render the content you want cited.
Retrieval splits documents into chunks and hands a model one at a time, without your headings or the paragraph above. A section opening "It also supports…" is clear in place and meaningless retrieved. Write every section to survive alone.
The GEO study (Aggarwal et al., KDD 2024) tested content edits across 10,000 queries and found adding statistics, quotable statements and cited sources produced the largest visibility gains. Concrete numbers beat confident adjectives, and it's one of the few claims here with a controlled study behind it.
Allowing a crawler in robots.txt doesn't mean it came. Server access logs are the only authoritative record. Analytics can't see bots, because bots don't run JavaScript. If a permissive file produces zero crawler hits, something upstream is blocking them.
Tutorials and tips for getting more from our geo studio.
A reference for the fifteen AI crawler tokens that decide your visibility in ChatGPT, Claude, Perplexity and Google, what each one actually does, and what blocking it costs you.
Allowing crawlers in robots.txt is permission, not proof. Your server access log is the only record of which AI bots arrived, what they fetched and what they got back.
Cloudflare now blocks AI crawlers at the firewall for new zones, before robots.txt is ever read. Here is why a permissive robots.txt proves nothing, and exactly where to look.
llms.txt is widely recommended and thinly evidenced. Here is what the convention proposes, what the major AI operators actually document, and when it is still worth twenty minutes.
A 2024 research paper tested content edits across 10,000 queries and found which ones increased citation by generative engines. Here is what it found, and what it did not.
Googlebot renders your page. GPTBot, ClaudeBot and PerplexityBot generally do not. Here is how to measure the gap between what users see and what AI systems receive.
Free, private and instant: everything runs in your browser.