My website needs a robots.txt file
Control what search engines crawl, and avoid the classic mistake of accidentally blocking your entire site.
A robots.txt file tells search-engine crawlers which parts of your site they may or may not crawl, and points them to your sitemap. Generate one with a robots.txt tool, set your allow/disallow rules, add the sitemap line, and validate it before uploading to your site root. Our generator and validator run in your browser.
4 tools for this
Each one is free, private and runs in your browser. No sign-up, no watermark.
Robots.txt Generator
Build a valid robots.txt with user-agent groups, allow/disallow rules, crawl-delay, sitemap and host directives, with live output, presets and lint warnings. 100% in your browser.
Robots.txt Validator
Paste your robots.txt to parse it into User-agent groups and catch the mistakes that block Google: a stray Disallow: /, rules before any User-agent, unknown directives and a missing sitemap. 100% in your browser.
XML Sitemap Generator
Turn a list of URLs into a valid sitemap.xml with per-URL changefreq, priority and lastmod, validated against the 50,000-URL limit and ready to submit to Google. 100% in your browser.
Meta Tag Generator
Generate perfect SEO meta tags, Open Graph and Twitter Card markup with a live Google snippet and social-card preview, copy the ready-to-paste <head> code, 100% in your browser.
How to fix it
Follow these steps and you'll have it sorted in a couple of minutes.
- 1
Generate your robots.txt
Use the generator to set which user-agents and paths are allowed or disallowed.
- 2
Add your sitemap
Include a 'Sitemap:' line pointing to your sitemap.xml so crawlers can find it.
- 3
Validate the rules
Run the Validator to confirm you haven't accidentally blocked pages you want indexed.
- 4
Upload to your root
Place it at yoursite.com/robots.txt. It must live at the domain root to take effect.
- 'Disallow: /'. This blocks your entire site from search engines. It's the most common and most damaging robots.txt mistake.
- Using robots.txt to hide private pages. It stops crawling, not access. Use authentication or noindex for real privacy.
- Placing it in a subfolder. Robots.txt only works at the domain root.
Frequently asked questions
More problems we solve
Ran into one of these too? Here's the fix.
- My website needs an XML sitemapHelp Google find every page. Generate a valid XML sitemap, submit it in Search Console, and link it from robots.txt.
- My site has no meta tags or preview when sharedA blank preview costs you clicks. Add the right tags so Google and social show a proper title, description and image.
- I want rich results / structured data for my pagesRich results earn more clicks. Add the right Schema.org markup so your pages qualify for enhanced listings.
Keep exploring
Related tools
Problems we solve
Definitions
From the blog
By file type
Common tasks
Ready? Start with Robots.txt Generator
Free, private and instant: everything runs in your browser.