Every AI Crawler User-Agent, and Whether to Allow It
A reference for the fifteen AI crawler tokens that decide your visibility in ChatGPT, Claude, Perplexity and Google, what each one actually does, and what blocking it costs you.
Guides, tips and updates on the tools you use every day: how-tos, deep dives and product news.
A reference for the fifteen AI crawler tokens that decide your visibility in ChatGPT, Claude, Perplexity and Google, what each one actually does, and what blocking it costs you.
Allowing crawlers in robots.txt is permission, not proof. Your server access log is the only record of which AI bots arrived, what they fetched and what they got back.
Cloudflare now blocks AI crawlers at the firewall for new zones, before robots.txt is ever read. Here is why a permissive robots.txt proves nothing, and exactly where to look.
llms.txt is widely recommended and thinly evidenced. Here is what the convention proposes, what the major AI operators actually document, and when it is still worth twenty minutes.
A 2024 research paper tested content edits across 10,000 queries and found which ones increased citation by generative engines. Here is what it found, and what it did not.
Googlebot renders your page. GPTBot, ClaudeBot and PerplexityBot generally do not. Here is how to measure the gap between what users see and what AI systems receive.
Retrieval systems split your content into chunks and hand a model one piece at a time, without your headings or the paragraph above. Here is what that does to ordinary writing.
One pasted.env file or rushed commit is all it takes to leak a live API key. The steps secrets end up in code, how to scan for them in seconds, and the habits that stop it happening again.
Our regex generator turns example strings into a working pattern with zero AI, and that's a feature, not a shortcut. A look under the hood, and an honest case for boring algorithms.
New tools and how-to articles land regularly. Follow along however you like. No inbox required.