Robots.txt Generator: Create a Valid robots.txt File Online
Add user-agent groups, choose which paths to allow or disallow, set a crawl-delay and point crawlers at your sitemap: then copy or download a spec-correct robots.txt. Linted live, generated entirely in your browser, no signup.
What is the robots.txt generator?
The Robots.txt Generator builds a spec-correct robots.txt from user-agent groups with allow/disallow rules, crawl-delay, sitemap and host directives. Live lint warnings catch a site-blocking Disallow: /, and presets cover allow-all, block-site, WordPress and block-AI-crawlers. Free and browser-based, so staging paths and unreleased sections stay private.
The GigAI Robots.txt Generator turns a few structured fields into a clean, standards-compliant robots.txt, the file that lives at your domain root and tells search-engine crawlers which parts of your site they may request. Create one or more user-agent groups (target every bot with `*`, or single out Googlebot, Bingbot, GPTBot and friends), add Allow and Disallow path rules, an optional Crawl-delay, one or more Sitemap URLs and an optional Host line. As you edit, the output updates instantly and a live linter watches for the mistakes that quietly de-index sites: a bare `Disallow: /` that blocks everything, a group with no rules, a Sitemap that isn't an absolute URL, or unknown directives. One-click presets get you started fast: Allow all, Block entire site, a WordPress-hardening template, and a block-AI-crawlers preset, and you can still hand-tune every line. When it's right, copy it or download robots.txt, drop it at https://yourdomain.com/robots.txt, and it takes effect on the next crawl. Because the whole thing is generated locally with no server and no crawl, your paths, staging rules and unreleased sections never leave your machine.
- Difficulty:
- Easy
- Typical time:
- ~30s
- Processing:
- 100% browser processing
Last updated
How to use the robots.txt generator
- 1
Start from a preset or blank group
Pick Allow all, Block entire site, WordPress or Block AI crawlers to prefill sensible rules, or begin with an empty `User-agent: *` group and build it up.
- 2
Add your allow / disallow rules
For each group, set the user-agent(s) and list the paths to Disallow (block) and Allow (carve back out), one path per line. Add a Crawl-delay only if a bot is hammering your server.
- 3
Point crawlers at your sitemap
Add your absolute sitemap URL(s) (e.g. https://example.com/sitemap.xml). Optionally set a Host line for your preferred domain.
- 4
Review the live lint
Read the warnings panel: confirm you aren't blocking the whole site, that every group has rules, and that your sitemap URL is absolute. Fix anything flagged.
- 5
Copy or download robots.txt
Copy the output or download robots.txt, then upload it to your domain root so it's reachable at https://yourdomain.com/robots.txt. It applies on the next crawl.
What Robots.txt Generator includes
Structured user-agent groups
Add as many groups as you need, one for every crawler with `*`, or targeted rules for Googlebot, Bingbot, GPTBot and others, each with its own Allow, Disallow and Crawl-delay.
Allow & Disallow path rules
Type the paths you want to block or open, one per line. Wildcards (`*`) and end-anchors (`$`) are passed through untouched so pattern rules like /*.pdf$ work as written.
Sitemap & host directives
List one or more absolute Sitemap URLs so every crawler discovers your sitemap, and optionally add a Host line, both emitted in the correct place, outside the agent groups.
Live lint & foot-gun warnings
Inline checks flag a site-wide `Disallow: /`, empty groups, non-absolute sitemap URLs and unknown directives as you type, so you never publish a robots.txt that hides your whole site.
One-click presets
Start from Allow all, Block entire site, a WordPress-hardening template or a block-AI-crawlers preset, then fine-tune. No memorising syntax from scratch.
Copy, download, share, undo
Copy the file, download a correctly-named robots.txt, share a link that reopens your rules, load an example, or undo/redo, all keyboard-driven and fully client-side.
Why use our robots.txt generator
Control crawl budget deliberately
Keep bots out of search, filter, cart and admin URLs so their limited crawl budget lands on the pages you actually want indexed, without touching a line of server config.
Avoid the catastrophic block
The live linter catches a stray `Disallow: /` before you ship it, so you never accidentally tell Google to drop your entire site from the index.
Correct syntax without the spec
You fill in plain fields and the tool emits the exact directive order and casing crawlers expect. No more debugging why a hand-typed rule is being ignored.
Private by design
Nothing is crawled or uploaded. Staging paths, internal sections and unreleased rules are generated locally and never leave your browser.
Built for the way you work
From quick one-off fixes to daily workflows, see how people put this tool to use.
- SEO specialist
Steer crawl budget on a large site
Block faceted-search, sort and pagination URLs while keeping category and product pages open, so Googlebot spends its budget on pages that can rank.
- Web developer
Lock down staging & admin paths
Generate a robots.txt that disallows /wp-admin/, /cart/, /checkout/ and internal tools, and add the sitemap line, in one pass instead of hand-editing the file.
- Site owner
Keep AI crawlers off your content
Use the block-AI-crawlers preset to disallow GPTBot, CCBot, ClaudeBot and other model crawlers while leaving Googlebot and Bingbot free to index for search.
- Agency / freelancer
Ship consistent robots.txt to clients
Produce a clean, linted robots.txt per client domain (including pre-launch staging rules) and share a pre-filled link so anyone can review it.
Supported formats
Accepts Form fields, and produces robots.txt and Text, all processed locally in your browser.
- Form fields
- robots.txt
- Text
Frequently asked questions
Use this in a workflow
See how the robots.txt generator fits into a step-by-step journey with related tools.
Recommended tools
XML Sitemap Generator
Turn a list of URLs into a valid sitemap.xml with per-URL changefreq, priority and lastmod, validated against the 50,000-URL limit and ready to submit to Google. 100% in your browser.
Meta Tag Generator
Generate perfect SEO meta tags, Open Graph and Twitter Card markup with a live Google snippet and social-card preview, copy the ready-to-paste <head> code, 100% in your browser.
Sitemap Validator
Paste your sitemap.xml and validate it instantly: well-formed XML, every <url> has a <loc>, the 50,000-URL limit, HTTPS locations and valid changefreq/priority. Catches the errors that make Google reject a sitemap. 100% in your browser.
URL Encoder
Percent-encode text and URLs instantly: pick component mode for query values or full-URL mode to preserve structure, with a per-line batch switch. 100% in your browser, nothing uploaded.
Robots.txt Validator
Paste your robots.txt to parse it into User-agent groups and catch the mistakes that block Google: a stray Disallow: /, rules before any User-agent, unknown directives and a missing sitemap. 100% in your browser.
AI Crawler Access Checker
Check whether GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Bingbot and every other AI crawler can read your site. Real RFC 9309 robots.txt matching with the deciding rule quoted, plus Cloudflare default-blocking detection. Free, instant, no signup.
Comparisons & guides
Go deeper on the concepts behind the robots.txt generator.
Common problems, solved
Hit a snag? Here are quick fixes for the issues people run into most.
My whole site disappeared from Google after adding robots.txt.
You almost certainly have `Disallow: /` under `User-agent: *`, which blocks every crawler from every path. Remove that line (or scope it to specific folders), re-upload robots.txt, then request re-crawling in Search Console. The linter here flags this before you publish.
A page I disallowed still shows up in search results.
robots.txt only blocks crawling, not indexing. A disallowed URL that's linked from elsewhere can still appear (often with no description). To keep a page out of the index, allow crawling and add a `<meta name="robots" content="noindex">` tag or an X-Robots-Tag header instead.
Crawlers seem to ignore my Crawl-delay.
Googlebot does not honour Crawl-delay at all. Set the crawl rate in Search Console instead. Bing, Yandex and some others do respect it. Only add Crawl-delay for the bots that use it, and keep the value modest so you don't starve legitimate crawling.
My rules don't seem to match the URLs I expected.
robots.txt matching is prefix-based and case-sensitive on the path. `Disallow: /Blog` won't match /blog. Use `*` for wildcards and `$` to anchor the end (e.g. `Disallow: /*.pdf$`). Test your final file with Google's robots.txt Tester after uploading.
Get the most out of it
Put robots.txt only at the domain root, https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own.
Always include at least one absolute Sitemap: line so every crawler, not just Google, can find your sitemap.
Use robots.txt to manage crawling, and meta robots / X-Robots-Tag to manage indexing. They solve different problems.
Be specific with Disallow paths. A trailing slash matters (/private/ blocks the folder, /private also matches /privateers).
Keep the file small and readable, group related rules, comment sections with `#`, and avoid contradictory Allow/Disallow lines.
What's new
Recent updates and improvements to the robots.txt generator.
Initial release, structured user-agent groups with Allow/Disallow/Crawl-delay, Sitemap and Host directives, live output, lint warnings, copy and download robots.txt.
Added one-click presets (Allow all, Block entire site, WordPress, Block AI crawlers), duplicate-group and empty-group detection, and shareable links.
Added multi-agent groups per block, keyboard shortcuts with an in-tool helper, and refined lint messaging for wildcard and anchored path rules.
Keep exploring
Related tools
Problems we solve
Definitions
From the blog
Explore categories
By file type
Common tasks
Your privacy is built in
Your robots.txt is generated entirely in your browser. There is no crawl and no upload. Paths, staging rules and unreleased sections are never stored, logged or transmitted, so you can safely draft rules for internal or pre-launch URLs. Remember that a published robots.txt is itself public, so never rely on it to hide truly sensitive URLs: block those with authentication instead.
- Runs in your browser
- No uploads
- Nothing stored
Ready to try the robots.txt generator?
Free, private and instant. Robots.txt Generator runs right in your browser.