Every GEO article tells you to allow AI crawlers. None of them can tell you whether the crawlers came. Your server's access log can, and it's the only source that knows. Drop an nginx or Apache log here and get per-bot visit counts, last-seen timestamps, the pages each crawler reads most, the HTTP statuses they received, and a daily trend: for GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Bingbot and every other AI token. Parsed entirely in this browser tab, because access logs contain your visitors' IP addresses.
The AI Crawler Log Analyzer parses nginx and Apache access logs in your browser to show which AI crawlers actually visited: per-bot request counts, first and last seen, most-fetched paths, HTTP status health and a daily trend. It also lists pages no AI crawler has ever fetched. Access logs contain visitor IP addresses, so nothing is uploaded and the IP column is discarded at parse time.
robots.txt tells you what crawlers are permitted to do. Logs tell you what they did. That gap is where most AI-visibility work goes wrong: a site can have a permissive robots.txt, a clean rendering setup and zero AI crawler traffic, because a firewall rule, a CDN, a WAF or a bot-management default is silently rejecting them upstream. This analyzer reads standard nginx and Apache combined/common-format logs and reports what actually happened, showing which of the tracked AI crawlers appeared, how many requests each made, when each was first and last seen, the paths each fetched most, the distribution of HTTP status codes they received (a bot getting 403s is worse than a bot not visiting), and a per-day activity trend so you can see whether crawling is growing, flat or stopped. It also inverts the question: paste your page URLs or sitemap and it lists the pages no AI crawler has ever fetched, which is usually a more actionable list than the ones they have. If zero AI hits appear, it says so bluntly and explains the three likely causes. The privacy architecture here is not a marketing line: access logs contain visitor IP addresses, so the file is read in your browser, the parser discards the IP column at parse time rather than storing it, and nothing is ever uploaded to any server.
Last updated
Typically /var/log/nginx/access.log or /var/log/apache2/access.log. Gunzip rotated files first (gunzip access.log.1.gz). A month of data gives the most useful trend.
The file is read locally in this tab. Nothing uploads. The parser drops the IP column as it reads. Files up to 200 MB are handled.
Total AI requests, how many distinct crawlers, the log's date range, then a card per bot with counts, dates, top paths, statuses and daily trend.
Paste your sitemap URLs or paths into the box at the bottom to get the never-fetched list.
No hits at all: check crawler access and your CDN. Hits with poor status codes: check what's rejecting them. Never-fetched pages: improve internal linking and sitemap coverage.
The file is read in this tab and the parser discards the IP column outright. It is never stored, displayed or transmitted. This is the one tool where 'no upload' is a legal and ethical requirement, not a feature.
Every AI crawler that appeared, with request counts, the date range it was active, and a daily trend chart showing whether its interest is growing or has stopped.
The top paths per bot, which reveal what a crawler thinks your site is about, and often that it's spending its budget on pages you don't care about.
A crawler receiving 403s or 404s is being turned away, which looks like traffic in a naive count. The 2xx/3xx success rate per bot separates 'visited' from 'served'.
Paste your URL list or sitemap and get the inverse report: the pages no AI crawler has ever touched. Usually the most actionable output in the whole tool.
If no AI crawler appears at all, the tool says so plainly and names the three likely causes: log window, robots/firewall blocking, or a CDN answering bots before your origin ever sees them.
Allowing crawlers in robots.txt is a permission, not an outcome. Logs are the only proof that permission turned into visits.
A permissive file plus zero hits points straight at a firewall, WAF or CDN rule: a diagnosis you cannot reach any other way.
Crawl coverage is uneven and heavily skewed to whatever is well-linked. The never-fetched list tells you where internal linking or sitemap work will pay off.
Uploading raw access logs to a third-party service means uploading your visitors' IP addresses. This tool makes that impossible by design.
From quick one-off fixes to daily workflows, see how people put this tool to use.
"Is ChatGPT reading my site?" has exactly one authoritative answer, and it's in your access log. Ten seconds to find out.
When robots.txt is clean but crawlers never arrive, the log proves the blocking is happening above the application, and the status codes narrow down where.
Per-bot top paths plus the never-fetched list turn AI crawl budget from an abstraction into a specific list of neglected URLs.
If you deliberately blocked training crawlers, the log shows whether they stopped: including whether Bytespider, which has a history of ignoring robots.txt, actually complied.
Boundaries stated plainly, with the right tool for each neighbouring job.
Accepts LOG and TXT, and produces Report, all processed locally in your browser.
Check whether GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bingbot and every other AI crawler can read your site. Real RFC 9309 robots.txt matching with the deciding rule quoted, plus Cloudflare default-blocking detection. Free, instant, no signup.
Generate a correct robots.txt policy for every AI crawler (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and 11 more) with per-bot allow/block toggles and a plain-language line on what each choice costs. Free, in your browser.
One URL in, one AI-visibility report out: crawler access with Cloudflare detection, JS rendering, citability scored with published methodology, retrieval chunks, answer snippets and schema. Each finding linked to the free tool that fixes it.
See your page the way AI crawlers see it: without JavaScript. Fetch the raw HTML, count the words that survive, detect client-rendering framework markers, and diff against the rendered DOM to list exactly which content is invisible to GPTBot, ClaudeBot and PerplexityBot.
Audit a page's JSON-LD against the seven schema types AI retrieval leans on, verify each one's load-bearing properties, and get every gap routed to the GigAI generator that fills it. A checker, not another generator.
Check whether every section of your page opens with a self-contained answer in 40 words or fewer. Flags throat-clearing intros, buried answers and openings that depend on missing context, with a rewrite target for each failure.
Hit a snag? Here are quick fixes for the issues people run into most.
The parser expects the nginx/Apache combined or common log format. Custom log_format directives that reorder or omit fields won't match. Either export in the default format or configure a matching one. The report tells you what proportion matched so you know how much you're missing.
Three usual causes: the log window predates AI crawling on your site. A firewall, WAF or Cloudflare's AI-bots setting is rejecting them before your origin, or a CDN is serving them from cache so requests never reach the origin log. Check the crawler access checker for the Cloudflare signal, then look at CDN-level logs.
It's being turned away, not served. Look at which statuses dominate: 403 usually means a WAF or bot rule, 404 means it's crawling URLs that no longer exist (often stale sitemap entries), 429 means rate limiting. Fix the cause, or the visits are worthless.
Gunzip it first (gunzip access.log.2.gz): compressed logs aren't parsed, and the tool will tell you so rather than failing silently. Concatenating several days into one file before dropping it gives a better trend.
A month of logs is the sweet spot: enough to see a trend, small enough to parse instantly.
User-agent strings can be spoofed. High-confidence verification means reverse-DNS checking the IP against the operator's documented ranges. This tool identifies by user-agent, which is right for measuring interest, not for security decisions.
Compare bots against each other rather than against an absolute target: if Bingbot visits 6,000 times and OAI-SearchBot 12, the ratio is the finding.
If you're behind a CDN, origin logs undercount badly: cached responses never reach your server. Pull CDN logs for the true picture.
Run this after changing robots.txt: crawlers re-read the file within about a day, so a week later the log shows whether your change had the effect you intended.
Recent updates and improvements to the ai crawler log analyzer.
Initial release: in-browser combined/common log parsing with IP discarded at parse time, per-bot hit counts, first/last seen, top paths, status-code health, 30-day trend bars, never-fetched page detection from a pasted URL list, and blunt zero-hit diagnosis.
Access logs contain your visitors' IP addresses, which is exactly why this tool never uploads them. The file is read in your browser, the IP field is discarded during parsing rather than stored, and no request carrying log data is ever made. Close the tab and nothing remains.
Free, private and instant. AI Crawler Log Analyzer runs right in your browser.
