Free Robots.txt Generator
Create a valid robots.txt file in seconds. Start from a preset, fine-tune allow/disallow rules per bot, add your sitemap, and download the file. No sign-up required.
100% Private
Instant
One-click Download
Works on Mobile
Configure Your Robots.txt
Start with a preset
Adds a
Sitemap: line at the end of the file. Recommended.Rules
Advanced options
Adds
Crawl-delay: — Bing and Yandex honour this, Google ignores it.
Clean output (no comments)
robots.txt
Where to upload: save this file as
robots.txt and upload it to the root of your domain so it is reachable at https://yoursite.com/robots.txt. It must be in the root directory — subfolders will not work.
What Is a robots.txt File?
A robots.txt file is a plain text file that lives in the root of your website. It tells search engine crawlers which parts of your site they are allowed to crawl — and which parts they should stay away from. It's part of the Robots Exclusion Protocol, a standard supported by Google, Bing, Yandex and every other major search engine.
When a crawler arrives at your site, it first checks https://yoursite.com/robots.txt. If the file doesn't exist, the crawler assumes it can crawl everything. If the file exists, the crawler reads the rules and respects them.
robots.txt syntax in 30 seconds
User-agent: *— applies the following rules to every crawler. Replace*with a specific bot name (likeGooglebot) to target it.Disallow: /path/— tells the bot not to crawl that path.Allow: /path/page.html— re-allows a path that a broader Disallow would have caught.Sitemap: https://yoursite.com/sitemap.xml— points crawlers to your sitemap. One per line, multiple allowed.Crawl-delay: 5— tells crawlers to wait 5 seconds between requests. Google ignores this; Bing and Yandex honour it.- Lines starting with
#are comments.
What robots.txt does NOT do
- It does not block indexing. A disallowed page can still appear in Google search if it's linked from elsewhere. To prevent indexing, use a
noindexmeta tag or anX-Robots-TagHTTP header. - It does not enforce security. A malicious scraper can simply ignore robots.txt. Never use it to "hide" private data — use authentication.
- It does not work in subfolders. The file must be at the root of the domain.
Common robots.txt mistakes
- Using
Disallow: /during development and forgetting to remove it before launch — this blocks Google entirely. - Forgetting to whitelist important resources like CSS and JS files — this can make Google see a broken page.
- Blocking
/wp-admin/with WordPress while still allowing/wp-admin/admin-ajax.php, which is needed by some frontend features. - Writing Windows-style CRLF line endings in some configurations — always save as plain UTF-8.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a plain text file placed in the root of your website that tells search engine crawlers which pages or sections they are allowed — or not allowed — to crawl. It is part of the Robots Exclusion Protocol.
Where do I upload robots.txt?
Upload it to the root of your domain so it is reachable at
https://yoursite.com/robots.txt. It must be in the root — it will not work in a subfolder.Does robots.txt block a page from Google?
No. robots.txt only tells crawlers not to fetch a page. Google can still index a URL it discovers elsewhere. To prevent indexing, combine robots.txt with a
noindex meta tag or an X-Robots-Tag header.How do I block all bots?
Use
User-agent: * on one line, then Disallow: / on the next. That blocks every crawler from crawling your entire site. Never do this on a live site — only on staging environments.Should I block AI crawlers like GPTBot?
It depends on your goals. If you want your content to appear in AI answers, allow them. If you want to keep your content out of AI training data, disallow them. Our "Block AI Bots" preset does this in one click.
Is this robots.txt generator free?
Yes — completely free, no sign-up, no limits. Everything runs in your browser and nothing is uploaded to any server.