Every AI Crawler User-Agent, and Whether to Allow It
A reference for the fifteen AI crawler tokens that decide your visibility in ChatGPT, Claude, Perplexity and Google, what each one actually does, and what blocking it costs you.
Every article tagged “geo” on GigAI Tools: guides, tips and how-tos.
6 articles in this collection.
A reference for the fifteen AI crawler tokens that decide your visibility in ChatGPT, Claude, Perplexity and Google, what each one actually does, and what blocking it costs you.
Allowing crawlers in robots.txt is permission, not proof. Your server access log is the only record of which AI bots arrived, what they fetched and what they got back.
Cloudflare now blocks AI crawlers at the firewall for new zones, before robots.txt is ever read. Here is why a permissive robots.txt proves nothing, and exactly where to look.
llms.txt is widely recommended and thinly evidenced. Here is what the convention proposes, what the major AI operators actually document, and when it is still worth twenty minutes.
Googlebot renders your page. GPTBot, ClaudeBot and PerplexityBot generally do not. Here is how to measure the gap between what users see and what AI systems receive.
Retrieval systems split your content into chunks and hand a model one piece at a time, without your headings or the paragraph above. Here is what that does to ordinary writing.
New tools and how-to articles land regularly. Follow along however you like. No inbox required.