Should You Block GPTBot and Other AI Crawlers in robots.txt?

Sooner or later someone on your team forwards you a blog post about AI companies training on your content, and asks whether they should add a few Disallow lines to robots.txt. It's a fair question, and the internet's answer is unhelpfully split between "block everything immediately" and "blocking is pointless anyway".

Both takes miss the part that actually matters: the AI bots hitting your site are not all doing the same job. Some collect text that may end up in training data. Others fetch your page in real time so an assistant can quote it and link to you. Block the wrong one and you quietly remove yourself from a traffic channel while gaining nothing. This guide walks through what each bot does, what you give up by blocking it, when blocking is genuinely the right call, and the exact rules to paste for each stance.

The two kinds of AI crawler

Almost every decision here gets easier once you separate the crawlers into two buckets.

Training crawlers

These gather content that may be used to train or improve a model. The value exchange is one-directional: your text goes in, nothing obviously comes back. This is the bucket people are usually thinking about when they say "I don't want AI stealing my content" — GPTBot, ClaudeBot and CCBot all live here.

Retrieval and citation crawlers

These fetch pages so an assistant can answer a question right now, usually with a visible link back to you. OAI-SearchBot builds the index behind ChatGPT's search results; ChatGPT-User fetches a page because a user asked ChatGPT to look at it; PerplexityBot indexes pages that Perplexity cites in its answers. Blocking these is much closer to blocking Googlebot than to opting out of training — you are removing yourself from a discovery surface, not protecting an asset.

The common mistake is treating both buckets as one line item. If your goal is "don't train on me, but do cite me", you need to block the first group and keep the second one open.

What each bot actually does

Here's the practical breakdown of the user-agents worth knowing, and what you lose by blocking each.

• GPTBot — OpenAI's crawler for content that may be used in model training. Blocking it is the clean way to opt out of training without touching ChatGPT's ability to cite you.
• OAI-SearchBot — builds the index behind ChatGPT's search. Block this and you're not in ChatGPT's results, full stop. This is the one people block by accident.
• ChatGPT-User — a live fetch triggered by a user pasting or asking about your URL. Blocking it means ChatGPT tells the user it can't read your page.
• ClaudeBot — Anthropic's crawler for training data. Anthropic also runs separate agents for user-triggered fetches (Claude-User) and search indexing (Claude-SearchBot), so the training/citation split applies here too.
• PerplexityBot — indexes pages so Perplexity can cite them. Perplexity is comparatively citation-heavy, so blocking it has a real visibility cost.
• Google-Extended — not a crawler at all. It's a control token that tells Google whether content it already crawled may be used for Gemini and related AI products. Blocking it has no effect on Googlebot, your crawl budget, or your rankings.
• Applebot-Extended — the same idea for Apple: it governs whether Apple may use your content for training, and is separate from Applebot, which powers Siri and Spotlight.
• CCBot — Common Crawl, an open web corpus that many datasets are built from. Blocking it stops future collection but does nothing about the snapshots already published.

A useful rule of thumb: the "-Extended" tokens are opt-outs with almost no downside, the training crawlers are a genuine policy choice, and the retrieval crawlers should be treated like search engines.

What blocking actually costs you

Three things are worth being clear-eyed about before you start writing Disallow lines.

Blocking is not retroactive. If GPTBot crawled you last year, that data is already collected. A rule added today affects future crawls only. This is the single biggest reason "block everything" is a weaker move than it feels — you're locking a door the visitors have already walked through.

robots.txt is a request, not enforcement. The major AI companies honour it, which is precisely why it works on them. Scrapers that ignore robots.txt were never going to read it, so blocking costs you visibility with the honest crawlers while doing nothing about the dishonest ones. If you need actual enforcement, that's a WAF, rate limiting or authentication — a different layer entirely.

You can't opt out of Google's AI Overviews this way. AI Overviews are part of Google Search and are governed by Googlebot, not Google-Extended. The only levers are the standard snippet controls (nosnippet, max-snippet) — which also shrink your normal search snippets — or leaving Google Search. Blocking Google-Extended covers Gemini and grounding, not Search.

When blocking is genuinely justified

None of the above means blocking is wrong. There are cases where it's clearly the right call:

• Paywalled or subscription content. If people pay for the text, giving it away to a training corpus undercuts the product. Block the training crawlers on those paths at minimum.
• Licensed or third-party material. If you don't own the rights, you may not have the right to let it be used for training in the first place.
• User-generated content and personal data. Forums, profiles and community posts contain other people's words. Their consent isn't yours to give.
• You're negotiating a content licence. Several publishers block first specifically to have something to license later. That's a commercial strategy, and a legitimate one.
• Bandwidth pressure. Aggressive crawling on a large site is a real cost. Rate limiting is usually the better tool, but blocking is defensible.

For most marketing sites, documentation, SaaS products and blogs, the calculus goes the other way: you publish content precisely so people find it, and AI assistants are now part of how they do. That's the reasoning behind our own robots.txt, which explicitly allows all eight of the bots above.

Copy-paste robots.txt blocks

Three coherent stances. Pick one, paste it, and don't mix and match without re-reading the section that follows.

Stance 1 — open to everything (recommended for most content-led sites):

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Named AI bots fall back to the * group, so this already allows them. Adding explicit Allow blocks per bot, as we do, changes nothing technically — it just documents the intent for the next person who edits the file.

Stance 2 — no training, yes citations (the option most people actually want):

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

OAI-SearchBot, ChatGPT-User and PerplexityBot are deliberately absent — they fall through to * and stay allowed, so you remain citable.

Stance 3 — block AI entirely:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Note that even here, the * group at the bottom keeps Googlebot, Bingbot and everything else working normally.

Mistakes we keep seeing

Named groups replace the wildcard — they don't stack. This is the rule that catches people out. If a crawler finds a group matching its own user-agent, it uses only that group and ignores User-agent: * completely. So writing User-agent: GPTBot followed by Disallow: /admin/ doesn't add a restriction on top of your general rules — it discards them for GPTBot, which is then free to crawl everything else you thought was disallowed.

Blocking Googlebot while aiming at Google's AI. Google-Extended and Googlebot are different tokens with completely different consequences. A Disallow: / under the wrong one deindexes your site from search.

Blocking OAI-SearchBot while hoping for ChatGPT citations. These are mutually exclusive. If you want to appear in ChatGPT's answers with a link, OAI-SearchBot must be allowed.

Assuming robots.txt hides anything. It's a public file that lists exactly which paths you consider sensitive. Never use it to protect private URLs.

Never checking. Files drift, deploys overwrite, someone copies a template from a Stack Overflow answer written in 2023. A stance you never verify isn't a stance.

How to verify your file matches your intent

Fetch https://yourdomain.com/robots.txt in a browser and read it end to end — it should be short enough to hold in your head. Check that each AI bot you care about appears in the group you think it does, and that your Sitemap: line is still there. If your site serves different files per environment, check production specifically.

For a faster answer across all eight bots at once, our AI-Friendly Checker fetches your live robots.txt during a scan and reports each AI crawler as allowed or blocked, alongside whether you publish an llms.txt. It applies the same most-specific-group rule the crawlers do, so it catches the "named group replaces the wildcard" trap that reading by eye often misses. And if your broader goal is to be cited rather than merely crawlable, our guide to Generative Engine Optimization covers the rest of the picture — server-side rendering, structured data and the signals that decide whether a model can actually use your page once it's allowed in.

Key takeaways

• Split the bots into training crawlers and retrieval crawlers before deciding anything — they cost you completely different things.
• GPTBot, ClaudeBot and CCBot are training crawlers. OAI-SearchBot, ChatGPT-User and PerplexityBot fetch pages to cite them; blocking those is closer to blocking Googlebot.
• Google-Extended and Applebot-Extended aren't crawlers — they're training opt-out tokens with no effect on search rankings or crawl budget.
• Blocking is neither retroactive nor enforcement. It stops future crawls by honest bots and does nothing about scrapers or data already collected.
• Blocking is justified for paywalled, licensed, user-generated or rights-encumbered content — and usually counterproductive for marketing sites and documentation.
• A named user-agent group replaces the * group entirely, which is how most accidental over- and under-blocking happens.
• Verify what's live, either by reading the file directly or by scanning with our AI-Friendly Checker.