robots.txt Generator
Build a correct robots.txt file without touching documentation, crawler rules, optional AI bot blocking, and your sitemap directive, ready to copy.
Build a correct robots.txt file without touching documentation, crawler rules, optional AI bot blocking, and your sitemap directive, ready to copy.
Save this as robots.txt and upload it to the root of your site so it's reachable at yoursite.com/robots.txt.
Choose your default crawler rule, add any paths you want blocked, decide whether AI training bots may access your content, and paste your sitemap URL. Click Generate and copy the finished file. Upload it to your site's root directory, it must be reachable at yoursite.com/robots.txt exactly, not in a subfolder.
The robots.txt file gives crawling instructions to well-behaved bots. It tells search engines which parts of your site to crawl and which to skip, and it points them to your sitemap. Two things it does not do: it doesn't remove pages from Google's index (blocked pages can still appear in results if other sites link to them, use a noindex tag for that), and it doesn't provide security (malicious bots ignore it entirely, so never rely on robots.txt to hide sensitive content).
There's a genuine trade-off. Allowing crawlers like GPTBot means your content can inform AI models, which increasingly means being mentioned or cited when people ask AI assistants questions in your topic area, a growing discovery channel. Blocking them protects your content from being used as training data but removes you from that channel. Search crawlers like Googlebot are unaffected either way; the toggles above only touch training-specific bots. For most sites trying to grow, leaving them open is the pragmatic call. For sites whose content is the product, blocking is reasonable.
Say you run a store and want search engines to index your products but not your cart, checkout, or account pages, while also keeping AI training bots out. Set the default rule to allow, add /cart, /checkout, and /account/ as blocked paths, check the AI training boxes you want, and paste your sitemap URL. The generator produces a file with a general Allow rule, three Disallow lines under the same User-agent block, separate blocks for each AI bot you opted to block, and a Sitemap line at the end. That's the entire file; nothing else needs to be added by hand.
After uploading, visit yoursite.com/robots.txt directly in a browser to confirm it loads as plain text, not as a 404 or your homepage. Then use Google Search Console's URL Inspection tool on a blocked path to confirm Google recognizes the block. If you recently changed the file, give Google a day or two to notice the update, crawlers cache robots.txt and don't fetch it on every single request.
These two mechanisms get confused constantly. robots.txt controls crawling: whether a bot is allowed to fetch a URL at all. A meta robots tag (placed in a page's head, like <meta name="robots" content="noindex">) controls indexing: whether a page that was crawled should appear in search results. A page can be crawlable but noindexed, or blocked from crawling but still indexed if enough external links point to it without Google ever reading the page itself. If your goal is genuinely removing a page from search results, a noindex tag is the reliable tool. If your goal is conserving crawl budget or keeping bots out of functional areas like checkout flows, robots.txt is correct. Our Meta Tag Generator can help you build the head tags that pair with your crawling rules.
The checkboxes above let you block GPTBot, Google-Extended, and CCBot without affecting search crawlers at all, but whether you should is a real decision with a genuine trade-off, not a default to accept blindly. Our guide on whether to block AI crawlers in robots.txt walks through the case for each choice and how to decide based on your site's actual traffic model.
A robots.txt file gives crawling instructions to search engine bots. It tells crawlers which parts of your site to access and which to skip, and points them to your sitemap. It must live at your site root, reachable at yoursite.com/robots.txt.
No. Blocking a page in robots.txt prevents crawling, but the page can still appear in search results if other sites link to it. To remove a page from the index, use a noindex meta tag instead.
It's a trade-off. Allowing AI training crawlers means your content can be cited by AI assistants, which is a growing discovery channel. Blocking them protects your content from training use. Search rankings are unaffected either way.
Yes. The Writstack robots.txt Generator is completely free with no account or sign-up required. It runs entirely in your browser.