Robots.txt Generator
Build a valid robots.txt, including opting out of AI training crawlers.
Block AI training crawlers
Your robots.txt
User-agent: * Disallow:
Save this as robots.txt in your site’s root, so it is reachable at yoursite.com/robots.txt. Note that robots.txt asks crawlers not to fetch a page — it does not keep a page out of search results if other sites link to it, and it is not a security measure. Use noindex to keep pages out of results, and real authentication to keep them private.
Build a robots.txt file by choosing which crawlers to address and which paths to allow or disallow, with the correct syntax generated for you — including the sitemap reference that belongs in every robots.txt and is regularly left out.
robots.txt lives at the root of your domain and is the first thing most crawlers request. Its syntax is small but unforgiving: one misplaced slash can deindex an entire site.
The directives
| Directive | Does |
|---|---|
| User-agent: * | Applies the following rules to all crawlers |
| Disallow: /admin/ | Asks crawlers not to request anything under /admin/ |
| Disallow: / | Asks crawlers not to request anything at all |
| Allow: /admin/public/ | Carves an exception out of a broader Disallow |
| Sitemap: https://…/sitemap.xml | Points crawlers at your sitemap — use an absolute URL |
| Crawl-delay: 10 | Honoured by Bing and others; ignored by Google |
The most specific matching rule wins in Google's implementation, not the first one listed.
The misunderstanding that causes real damage
robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search results — Google may index it from inbound links alone, showing the URL with no description because it was never allowed to read the page.
Worse, blocking a page in robots.txt prevents Google from seeing a noindex tag on that page. So the standard mistake — adding both a Disallow and a noindex to remove a page — guarantees the page stays indexed, because the crawler is never permitted to read the instruction telling it to leave.
- To keep a page out of search results: allow crawling and use a noindex meta tag or X-Robots-Tag header.
- To reduce crawl load on infinite or worthless URL spaces (faceted filters, session IDs): use robots.txt.
- To protect anything private: use authentication. robots.txt is a public file that reads as a directory of what you'd rather people didn't look at.
- Never block your CSS or JavaScript. Google renders pages, and blocking assets makes your site look broken to the crawler.
Before you publish it
- Place the file at the domain root — https://example.com/robots.txt. A robots.txt in a subdirectory is ignored entirely.
- Check that paths start with a slash and that directory rules end with one.
- Confirm you haven't left 'Disallow: /' in place from a staging environment. This is the single most expensive typo in SEO.
- Add the absolute URL of your sitemap.
- Test it in Google Search Console's robots.txt report before and after deploying.
- Re-check it after any site migration or platform change, which is exactly when a staging file tends to escape into production.
Frequently asked questions
- Does robots.txt stop a page from appearing in Google?
- No. It stops crawling, not indexing. A blocked page can still be listed from inbound links, typically with no description. To keep a page out of results, allow crawling and use a noindex tag.
- Why is my noindex tag being ignored?
- Almost certainly because the page is also blocked in robots.txt. The crawler can't fetch the page, so it never sees the noindex instruction. Remove the Disallow and let it read the tag.
- Can I use robots.txt to hide private content?
- No — it's a publicly readable file that effectively advertises the paths you'd prefer people didn't visit. Anything genuinely private needs authentication.
- Do all crawlers obey robots.txt?
- Well-behaved ones do; it's a convention, not an enforcement mechanism. Scrapers and malicious bots ignore it entirely.
- Should I block my CSS and JavaScript?
- No. Google renders pages to evaluate them, and blocking assets makes your site appear broken or unstyled to the crawler, which can hurt how it's assessed.
- Where exactly does the file go?
- At the root of the domain: https://example.com/robots.txt. Subdomains need their own file, and a robots.txt anywhere other than the root is ignored.