Robots.txt Generator
Generate robots.txt file to control search engine crawling
Robots.txt Generator builds a correctly-formatted robots.txt file that tells search engine crawlers which parts of your site to crawl or skip, and points them to your sitemap — no need to memorize the exact syntax rules crawlers expect.
Write it like you're telling a person — one instruction per line, plain English. Leave it blank if you want the whole site crawlable.
Optional, but it helps crawlers find every page faster. You can also just mention it above ("Sitemap: https://...").
What we understood
How to use Robots.txt Generator
- 1
Choose whether to allow or block specific crawlers and paths.
- 2
Add your sitemap URL so crawlers can discover it automatically.
- 3
Review the generated robots.txt in the preview panel.
- 4
Download the file and upload it to your site's root directory (e.g. yoursite.com/robots.txt).
Everything you need to know about robots.txt
What is robots.txt?
robots.txt is a plain text file at the root of a domain (yoursite.com/robots.txt) that tells search engine crawlers which parts of the site they're allowed to crawl. It's a voluntary instruction, honored by well-behaved crawlers like Googlebot, not an access control mechanism — it can't stop anyone from actually visiting a URL.
What robots.txt can — and can't — do
This is the single most common misunderstanding: robots.txt does not reliably keep a page out of search results. It stops crawling, but a page that's blocked from crawling can still appear in search results (usually with no description) if other sites link to it — because Google already knows the URL exists, it just can't read its content. If you need a page fully excluded from search results, use a `noindex` meta tag on the page itself instead, which requires the page to be crawlable so the crawler can actually see that instruction.
The directives, explained
| Directive | What it does |
|---|---|
| User-agent | Which crawler the following rules apply to (`*` means all) |
| Disallow | A path the crawler shouldn't crawl |
| Allow | An exception to a broader Disallow rule |
| Sitemap | Points crawlers to your XML sitemap's location |
robots.txt vs. noindex vs. sitemap
| Purpose | |
|---|---|
| robots.txt | Controls what gets crawled |
| noindex meta tag | Controls what gets shown in search results (page must still be crawlable) |
| Sitemap | Helps crawlers discover and prioritize pages faster |
A mistake that can accidentally de-index a whole site
A stray `Disallow: /` (blocking the entire site) is a surprisingly common accident — often left over from a staging environment's robots.txt that got deployed to production by mistake. Always check a live site's robots.txt after deployment, not just once during setup.
Related SEO Tools
Meta Tag Generator
Generate complete HTML meta tags for any webpage
Sitemap Generator
Generate XML sitemaps from a list of URLs
Open Graph Generator
Generate Open Graph tags for social media sharing previews
Twitter Card Generator
Generate Twitter Card meta tags for rich link previews
Canonical URL Generator
Generate canonical link tags to prevent duplicate content
Schema Generator
Generate JSON-LD structured data for rich search results
Frequently Asked Questions
Where does robots.txt need to live?
At the root of your domain — e.g. https://yoursite.com/robots.txt — not in a subfolder, or crawlers won't find it.
Does robots.txt stop a page from being indexed?
Not reliably — it stops crawling, but a blocked page can still appear in search results if it's linked from elsewhere. Use a noindex meta tag if you need a page fully out of search results.
Can I block specific bots only?
Yes — target individual user-agents (like a specific AI crawler) while still allowing Googlebot and others.
Should every site have a robots.txt file?
It's not required, but having one — even a permissive one that just points to your sitemap — is standard practice and gives you explicit control instead of relying on crawler defaults.
What's the most common robots.txt mistake?
A stray "Disallow: /" that blocks the entire site — often a staging-environment robots.txt accidentally deployed to production. Always check a live site's robots.txt after deploying.
Does robots.txt affect existing indexed pages?
Blocking a page after it's already indexed doesn't remove it from search results by itself — it just stops future crawling. Use a noindex tag (on a page crawlers can still reach) to actually get a page removed.