I don't know whether AI crawlers are actually visiting my site
Allowing a crawler in robots.txt is permission, not proof. Only your logs show whether the crawlers came.
Your server's access log is the only authoritative record, because AI crawlers don't run JavaScript and are therefore invisible to analytics tools like GA4. Export an nginx or Apache log and search it for crawler user-agents, or parse it in your browser to get per-bot visit counts, last-seen dates and the pages each one fetched.
4 tools for this
Each one is free, private and runs in your browser. No sign-up, no watermark.
AI Crawler Log Analyzer
Drop a server access log and see which AI crawlers actually visit: per-bot hit counts, first and last seen, most-fetched pages, status-code health, daily trend, and the pages no AI crawler has ever touched. Parsed in your browser: logs contain visitor IPs and never leave your device.
AI Crawler Access Checker
Check whether GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bingbot and every other AI crawler can read your site. Real RFC 9309 robots.txt matching with the deciding rule quoted, plus Cloudflare default-blocking detection. Free, instant, no signup.
GigAI GEO Audit
One URL in, one AI-visibility report out: crawler access with Cloudflare detection, JS rendering, citability scored with published methodology, retrieval chunks, answer snippets and schema. Each finding linked to the free tool that fixes it.
AI Robots.txt Generator
Generate a correct robots.txt policy for every AI crawler (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and 11 more) with per-bot allow/block toggles and a plain-language line on what each choice costs. Free, in your browser.
How to fix it
Follow these steps and you'll have it sorted in a couple of minutes.
- 1
Export an access log
Usually /var/log/nginx/access.log or /var/log/apache2/access.log. Gunzip rotated files first. About a month of data gives a useful trend.
- 2
Parse it locally
Drop the file into the Log Analyzer. It runs in your browser and discards the IP column as it reads, because logs contain visitor IP addresses.
- 3
Read the per-bot cards
Each crawler shows request counts, first and last seen, most-fetched pages, status codes and a daily trend.
- 4
Check the status codes
A bot receiving 403s is being turned away rather than served. A high hit count with a low success rate is worse than no visits.
- 5
Find the never-fetched pages
Paste your sitemap URLs to list the pages no AI crawler has ever touched, which is usually the most actionable output.
- Using analytics to measure bot traffic. Bots don't run JavaScript, so GA4 and similar tools are structurally blind to them.
- Uploading raw logs to an online analyzer. Access logs contain your visitors' IP addresses, which is personal data. Parse them locally instead.
- Reading zero hits as proof of blocking. If you use a CDN, crawler requests may be served from edge cache and never reach your origin log at all.
- Trusting the user-agent for security decisions. Anyone can send any user-agent. Verify with reverse DNS against the operator's published ranges if it matters.
Frequently asked questions
More problems we solve
Ran into one of these too? Here's the fix.
- ChatGPT and other AI tools can't see my websiteWork through the three causes in order. Access first, rendering second, content last, because fixing content on a page crawlers can't reach changes nothing.
- I don't know whether to allow or block AI crawlersThere's no universally right answer on training, but there is a wrong one: blocking the search bots by accident while trying to block training.
- AI tools can read my page but never quote itBeing readable isn't the same as being quotable. These checks measure whether there's anything on the page a model could safely repeat.
Keep exploring
Related tools
Problems we solve
From the blog
By file type
Next steps
Ready? Start with AI Crawler Log Analyzer
Free, private and instant: everything runs in your browser.