Robots.txt Validator & Tester. Check Your robots.txt Online
Paste your robots.txt and instantly see how crawlers read it: every User-agent group, its Allow and Disallow rules, and a plain-English warning the moment a directive would block your whole site, sit in the wrong place, or go unrecognised. Parsed live and entirely in your browser. No crawl, no upload, no signup.
What is the robots.txt validator?
The Robots.txt Validator parses a pasted robots.txt into User-agent groups the way crawlers do, flagging a site-blocking Disallow: /, rules placed before any User-agent, misspelled directives, non-numeric crawl-delays and a relative or missing Sitemap. Free and browser-based, reading only what you paste, so pre-launch drafts never leave your machine.
The GigAI Robots.txt Validator parses a pasted robots.txt exactly the way search-engine crawlers group it (a run of User-agent lines followed by their Allow, Disallow and Crawl-delay rules) and then lints the result for the errors that quietly cost you traffic. It flags the single most damaging mistake, a lone `Disallow: /` that blocks the entire site from that crawler, and highlights groups that have no rules at all (and therefore allow everything). It catches Allow/Disallow lines that appear before any User-agent, unknown or misspelled directives (a stray `Dissallow` or `User-Agents` does nothing), Crawl-delay values that aren't numbers, and Sitemap lines that use a relative path instead of the absolute URL crawlers require. It also warns when there is no Sitemap directive at all, since that is the simplest way to help engines discover your sitemap.xml. Unlike checkers that fetch your live domain from a server, this reads only the text you paste, so it works on staging files, pre-launch drafts and copies you are editing before you deploy, none of which leave your machine. Paste, read the grouped breakdown and the warnings, fix the flagged lines, and ship a robots.txt you can trust before Google ever sees it.
- Difficulty:
- Easy
- Typical time:
- ~30s
- Processing:
- 100% browser processing
Last updated
How to use the robots.txt validator
- 1
Paste your robots.txt
Copy the contents of your file (usually at https://example.com/robots.txt) and paste it in, or drop a .txt file. Comments (#) and blank lines are handled just like a crawler handles them.
- 2
Read the grouped breakdown
See each User-agent group with its Allow, Disallow and Crawl-delay rules, plus the Sitemap and Host lines pulled out separately: the exact rule set crawlers apply.
- 3
Review the warnings
Work through the flagged issues: a site-wide Disallow: /, empty groups, unknown directives, misplaced rules, a missing or relative Sitemap. Each points at a line number.
- 4
Fix and re-paste
Correct the flagged lines in your editor and paste again to confirm it's clean. When it validates, deploy the file to your domain root.
- 5
Confirm in Search Console
After deploying, use Google Search Console's robots.txt report and URL Inspection to verify important URLs are Allowed and the file is being fetched.
What Robots.txt Validator includes
Parsed the way crawlers group it
See every User-agent group with its Allow, Disallow and Crawl-delay rules laid out separately: exactly how Googlebot and Bingbot bucket consecutive User-agent lines into one rule set.
Catches the site-killing Disallow: /
The moment a group blocks your whole site with a bare Disallow: /, it's flagged in red, the single most common reason pages vanish from Google after a launch or migration.
Unknown & misspelled directives
A stray Dissallow, User-Agents or Noindex does nothing at all. The linter surfaces every directive crawlers won't recognise, with the exact line number.
Sitemap & absolute-URL checks
Warns when there's no Sitemap directive to help discovery, and when a Sitemap line uses a relative path instead of the absolute https:// URL the protocol requires.
Line-numbered warnings
Every issue points at the offending line (rules before a User-agent, non-numeric Crawl-delay, empty groups) so you fix the file, not guess at it.
100% in your browser
No crawler, no fetch, no server. Your robots.txt, including staging and pre-launch drafts, is parsed locally and never uploaded, stored or logged.
Why use our robots.txt validator
Stop accidentally blocking Google
A single misplaced Disallow: / can deindex an entire site. Validating before you deploy catches it while it's still a one-character fix, not a traffic emergency.
Understand what your file actually says
robots.txt grouping is unintuitive: blank lines start new groups, consecutive User-agents share one. The grouped view shows the real rule set each crawler sees.
Fix silent, invisible mistakes
Misspelled directives and rules before a User-agent don't error: they're simply ignored. The linter makes those silent failures visible before they cost you crawling.
Validate drafts privately
Because nothing is fetched or uploaded, you can check a staging robots.txt or an edit-in-progress that isn't live yet without exposing it to anyone.
Built for the way you work
From quick one-off fixes to daily workflows, see how people put this tool to use.
- SEO specialist
Diagnose 'Blocked by robots.txt'
When Search Console reports pages blocked by robots.txt, paste the file to see which Disallow rule and which User-agent group is catching those URLs.
- Web developer
Pre-launch safety check
Staging sites ship with Disallow: / to stay out of the index. Validate before go-live so that block is removed and the production file actually allows crawling.
- DevOps / platform
Verify a generated robots.txt
When a framework or CDN emits robots.txt from config, paste the rendered output to confirm the directives, User-agents and sitemap line came out as intended.
- Agency / freelancer
Audit a client's file fast
Drop a client's robots.txt in for an instant read on whether crawling is open, over-restricted, or riddled with directives that do nothing.
Supported formats
Accepts TXT and robots.txt, and produces Report, all processed locally in your browser.
- TXT
- robots.txt
- Report
Frequently asked questions
Recommended tools
XML Sitemap Generator
Turn a list of URLs into a valid sitemap.xml with per-URL changefreq, priority and lastmod, validated against the 50,000-URL limit and ready to submit to Google. 100% in your browser.
Sitemap Validator
Paste your sitemap.xml and validate it instantly: well-formed XML, every <url> has a <loc>, the 50,000-URL limit, HTTPS locations and valid changefreq/priority. Catches the errors that make Google reject a sitemap. 100% in your browser.
Meta Tag Analyzer
Paste a page's HTML to audit its title, description, canonical, robots, Open Graph and Twitter Card tags: with a length score, missing/duplicate checks and a live search & social preview, 100% in your browser.
Canonical URL Generator
Build a clean rel=canonical tag from any URL: normalise the host, path and query, strip tracking parameters, and copy the ready-to-paste <link> tag, HTTP header and framework snippets, 100% in your browser.
Robots.txt Generator
Build a valid robots.txt with user-agent groups, allow/disallow rules, crawl-delay, sitemap and host directives, with live output, presets and lint warnings. 100% in your browser.
AI Crawler Access Checker
Check whether GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bingbot and every other AI crawler can read your site. Real RFC 9309 robots.txt matching with the deciding rule quoted, plus Cloudflare default-blocking detection. Free, instant, no signup.
Comparisons & guides
Go deeper on the concepts behind the robots.txt validator.
Common problems, solved
Hit a snag? Here are quick fixes for the issues people run into most.
Search Console says my pages are 'Blocked by robots.txt' but I don't see why.
Paste the file here and read the grouped view. Remember crawlers match the most specific User-agent group, and a bare Disallow: / in the group that matches Googlebot blocks everything. Also check for a path prefix like Disallow: /blog that catches more URLs than you expected.
I added Noindex: / to robots.txt but the page still ranks.
Noindex is not a valid robots.txt directive, the linter flags it as unknown, and crawlers ignore it. To keep a page out of the index, remove any Disallow that blocks it and add a <meta name="robots" content="noindex"> tag (or an X-Robots-Tag header) so the crawler can read the page and see the noindex.
My Sitemap line is being flagged.
The Sitemap directive must be an absolute URL, e.g. Sitemap: https://example.com/sitemap.xml, not a relative path like /sitemap.xml. Put it on its own line. It applies site-wide regardless of which User-agent group it sits near.
One of my groups shows a warning that it 'allows everything'.
A User-agent group with no Allow or Disallow rules imposes no restrictions, so that crawler can access the whole site. That's often intended, but if you meant to restrict it, add the Disallow rules under that User-agent line.
Get the most out of it
robots.txt controls crawling, not indexing. A blocked URL can still appear in results without a snippet. Use a meta robots noindex tag (on a crawlable page) to keep something out of the index.
Blank lines start a new group. Keep each User-agent and its rules together with no empty line between them, or the rules can detach from the agent you intended.
List the Sitemap as an absolute URL on its own line. It's read site-wide and is the easiest way to help every crawler discover your sitemap.xml.
Test the real production file at https://example.com/robots.txt, not a local draft. The file crawlers obey is the one served at your domain root.
Disallow is a prefix match: Disallow: /admin also blocks /administrator. Be precise, and use Allow to carve out exceptions within a blocked path.
What's new
Recent updates and improvements to the robots.txt validator.
Initial release, paste-first robots.txt parser and linter: User-agent grouping, Allow/Disallow breakdown, site-wide Disallow detection, unknown-directive and missing-sitemap warnings, all in-browser.
Added line-numbered warnings, relative-sitemap and non-numeric Crawl-delay checks, drag-and-drop .txt import, and a grouped rules view with Sitemap/Host pulled out.
Added a scored summary, ready-made example files (clean, blocks-everything, messy), keyboard shortcuts and copy of the report.
Keep exploring
Related tools
Problems we solve
Definitions
From the blog
Explore categories
By file type
Common tasks
Your privacy is built in
Your robots.txt is parsed and linted entirely in your browser. There is no crawler and no upload. Nothing is stored, logged or transmitted, so you can safely validate staging, pre-launch or edit-in-progress files. Close the tab and every trace is gone.
- Runs in your browser
- No uploads
- Nothing stored
Ready to try the robots.txt validator?
Free, private and instant. Robots.txt Validator runs right in your browser.