Should You Block AI Crawlers in robots.txt?

By Stephanie  ·  Last updated July 31, 2026

Since late 2023, a new category of bot has been showing up in server logs: crawlers that exist purely to gather training data for AI models. GPTBot, Google-Extended, CCBot, and a growing list of others behave differently from Googlebot in one important way. Blocking Googlebot costs you search visibility. Blocking an AI training crawler costs you nothing measurable today, and possibly something in the future. Whatever you decide, our free robots.txt generator makes it a thirty-second change rather than something that requires editing raw config files by hand. That asymmetry is why the decision deserves more thought than a reflexive allow or block.

Two different jobs, one file

It helps to separate what robots.txt actually controls into two categories. Search crawlers like Googlebot and Bingbot fetch your pages to build the index that search results come from. Block them and you disappear from search, full stop. AI training crawlers fetch your pages for a completely different purpose: feeding the dataset that trains the next version of a language model. Blocking them has zero effect on whether your site shows up in Google or Bing. The two categories are unrelated technically, they just happen to be governed by the same file.

The case for allowing them

When a language model is trained on your content, two things become possible. First, the model may reproduce your ideas, phrasing, or facts when someone asks it a related question, with or without attribution depending on the platform. Second, and more useful to a site owner, some AI products now cite live sources rather than relying purely on training data, tools like Perplexity, and increasingly Google's own AI Overviews, fetch and quote from real pages in response to a query. Being crawlable is a prerequisite for being cited that way. If your traffic strategy includes AI-driven discovery, and for most content sites in 2026 it increasingly should, blocking the crawlers that make that discovery possible works against your own goal.

The case for blocking them

The counterargument is straightforward: your writing took real time and skill to produce, and training crawlers use it without payment, credit, or consent in the traditional sense. Some publishers, particularly ones whose business model depends on the content itself being scarce, such as subscription journalism, view this as a direct competitive threat, a model trained on their archive can eventually answer the same questions their content was written to answer, without sending anyone to the original page. For that category of site, blocking training crawlers is a defensible, even necessary, business decision.

What the decision actually depends on

The right call comes down to what your site's traffic model looks like. If your business depends on people never getting your content's substance anywhere except your page, subscription content, exclusive research, licensed data, blocking makes sense. If your business depends on people finding you at all, which describes most free tools, blogs, and marketing sites, being discoverable through every available channel, including AI assistants, usually outweighs the training concern. A brand new site with no existing authority has an even stronger case for staying open: the priority is being found by anyone, anywhere, and turning away a growing discovery channel before you have traffic to spare is a cost most new sites can't afford.

A middle path: block training, allow citation

Not every AI bot fits neatly into "training" or "citation." Some crawlers do both. Where the distinction is clear, a reasonable middle ground exists: block bots that are purely building training datasets (CCBot, the crawler behind the Common Crawl dataset used by many models) while leaving open bots associated with live-citation products. In practice this requires checking each bot's stated purpose individually, since the same company sometimes operates separate crawlers for separate purposes. It's more maintenance than a blanket rule, but it lets you participate in AI-driven discovery while opting out of pure dataset harvesting.

How to implement whichever you choose

Blocking a specific bot in robots.txt takes three lines: a User-agent line naming the bot, a Disallow line with a single slash, and nothing else needed for that block. Multiple bots each get their own three-line block. The default rule for all other crawlers, including search engines, stays separate and unaffected. Nothing about this decision requires touching your search visibility at all, which is exactly why it's worth making deliberately rather than defaulting to whatever a template happened to include.

Revisit the decision periodically

This is not a permanent setting. As AI products evolve and as your own traffic sources become clearer through analytics, revisit the choice every few months. A site that started open because it needed discovery might reasonably shift to blocking training bots once organic search traffic is established. A site that started cautious might open up once it sees measurable referral traffic from AI assistants. The file takes thirty seconds to update whenever the calculus changes.

What to watch in your analytics

Most analytics platforms now break out referral traffic by source, and it's worth checking periodically whether visits attributed to chatgpt.com, perplexity.ai, or similar AI referrers are showing up at all. A site with zero AI referral traffic after several months has less to lose by blocking training crawlers, since the discovery channel isn't materializing regardless. A site seeing a meaningful and growing share of visits from AI sources has a much stronger case for staying open, since the channel is demonstrably working, and the decision becomes evidence-based rather than a guess made once at launch and never revisited.

Frequently asked questions