Every AI Crawler User-Agent, and Whether to Allow It
A reference for the fifteen AI crawler tokens that decide your visibility in ChatGPT, Claude, Perplexity and Google, what each one actually does, and what blocking it costs you.
Browse every GEO Studio article: guides, tips and how-tos from the GigAI Tools team.
7 articles in this collection.
A reference for the fifteen AI crawler tokens that decide your visibility in ChatGPT, Claude, Perplexity and Google, what each one actually does, and what blocking it costs you.
Allowing crawlers in robots.txt is permission, not proof. Your server access log is the only record of which AI bots arrived, what they fetched and what they got back.
Cloudflare now blocks AI crawlers at the firewall for new zones, before robots.txt is ever read. Here is why a permissive robots.txt proves nothing, and exactly where to look.
llms.txt is widely recommended and thinly evidenced. Here is what the convention proposes, what the major AI operators actually document, and when it is still worth twenty minutes.
A 2024 research paper tested content edits across 10,000 queries and found which ones increased citation by generative engines. Here is what it found, and what it did not.
Googlebot renders your page. GPTBot, ClaudeBot and PerplexityBot generally do not. Here is how to measure the gap between what users see and what AI systems receive.
Retrieval systems split your content into chunks and hand a model one piece at a time, without your headings or the paragraph above. Here is what that does to ordinary writing.
New tools and how-to articles land regularly. Follow along however you like. No inbox required.