What is robots.txt?
A plain-English definition, and the tools that work with it.
robots.txt is a plain-text file at a site's root that tells search-engine crawlers which URLs they may or may not crawl. It's used to keep crawlers out of admin, duplicate or low-value areas and to point them at the sitemap, but it controls crawling, not privacy.
- It must live at the domain root (yoursite.com/robots.txt) to take effect.
- 'Disallow: /' blocks your entire site from search engines, the most common and damaging mistake, so always validate.
- robots.txt stops crawling, not access or indexing. Use noindex or authentication for genuinely private pages.
Tools for robots.txt
Free, private and in your browser. No sign-up.
Robots.txt Generator
Build a valid robots.txt with user-agent groups, allow/disallow rules, crawl-delay, sitemap and host directives, with live output, presets and lint warnings. 100% in your browser.
Robots.txt Validator
Paste your robots.txt to parse it into User-agent groups and catch the mistakes that block Google: a stray Disallow: /, rules before any User-agent, unknown directives and a missing sitemap. 100% in your browser.
XML Sitemap Generator
Turn a list of URLs into a valid sitemap.xml with per-URL changefreq, priority and lastmod, validated against the 50,000-URL limit and ready to submit to Google. 100% in your browser.
Meta Tag Generator
Generate perfect SEO meta tags, Open Graph and Twitter Card markup with a live Google snippet and social-card preview, copy the ready-to-paste <head> code, 100% in your browser.
robots.txt problems we solve
Hit one of these? Here's the fix.
Frequently asked questions
Related terms
- XML sitemapAn XML sitemap is a file listing a website's URLs so search engines can discover and index them efficiently. It doesn't guarantee ranking, but it helps Google and Bing find every important page (especially on large or new sites) and speeds up indexing of fresh content.
- Structured data (Schema)Structured data is machine-readable markup (usually Schema.org JSON-LD) that describes a page's content to search engines. It makes a page eligible for rich results (FAQ drop-downs, star ratings, recipe cards, article listings) that stand out in search and can lift click-through.
Keep exploring
Related tools
Problems we solve
Definitions
Try Robots.txt Generator
Free, private and instant. Everything runs in your browser.