The AI crawlers are not one thing, and that distinction matters

A lot of advice published recently told site owners to block AI bots. The advice treated them as a single group. They are not, and the difference decides whether you can be cited in an answer at all.

Run the free check

Each major AI company runs separate crawlers for training and for search. OpenAI documents GPTBot for training and OAI-SearchBot for surfacing sites in ChatGPT's search features. Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User. Perplexity separates PerplexityBot and Perplexity-User, and states neither is used for model training. Blocking the search agents removes you from the answers, which is usually the opposite of what the site owner intended.

Why this page exists

In the last two years a great deal was written arguing that AI systems consume your content and return nothing, and that you should block them. Many site owners agreed, and copied a robots.txt block from one of those posts.

The blocks that circulated were mostly written as a training opt-out. But they were often applied to every AI-related user agent the author could name, and that group includes the agents responsible for finding pages to cite in answers.

The result is a large number of sites that opted out of being recommended while believing they had opted out of being trained on. If you copied a robots.txt from anywhere, this is worth ten minutes today.

OpenAI

OpenAI documents four separate user agents, each with a stated purpose.

  • OAI-SearchBot is used to surface websites in ChatGPT's search features. This is the one that matters for being cited. Allow it.
  • GPTBot crawls content that may be used to train the foundation models. This is the genuine training opt-out, and whether you allow it is a real decision with reasonable answers either way.
  • ChatGPT-User visits a page when a user asks ChatGPT or a Custom GPT a question. Blocking it means a person who explicitly asked about your page cannot have it fetched.
  • OAI-AdsBot validates the safety of pages submitted as ads on ChatGPT, which only concerns you if you advertise there.

Anthropic

Anthropic documents three crawlers with the same separation of purpose.

ClaudeBot is the training crawler. Claude-SearchBot indexes content for search results. Claude-User retrieves pages when a user directs Claude to look at something. All three honour robots.txt, and they can be controlled independently, so blocking ClaudeBot does not prevent Claude-SearchBot from indexing you.

Two older agent names, Claude-Web and Anthropic-AI, are deprecated. If your robots.txt names those and nothing else, it is out of date and is not doing what you think it is doing.

Perplexity and Google

Perplexity documents PerplexityBot, which surfaces and links websites in Perplexity search results, and Perplexity-User, which visits pages when a user asks a question so that responses can cite sources. Perplexity states that neither is used to crawl content for training foundation models. On that documentation, there is no training reason to block either, and blocking them removes you from a surface where new products do comparatively well.

Google is the case most often misunderstood. Google-Extended is a separate token controlling whether crawled content may be used to train and ground Gemini models. Google's documentation states directly that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal in Google Search.

The practical consequence is that Google-Extended is a clean training opt-out with no search cost. Your eligibility to appear in Google's AI features comes through the ordinary Search index, which is Googlebot's job, so the decision about Gemini training is genuinely separate from the decision about being findable.

A robots.txt that reflects the distinction

There is no single correct file, because the training question is a real choice. But the shape below expresses the position most small products actually hold, which is happy to be found and cited, undecided or unwilling about being training data.

Allow every search and user-triggered agent: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, and of course Googlebot and Bingbot.

Then make a deliberate decision, once, about the training agents: GPTBot, ClaudeBot, and Google-Extended. Allowing them may increase the chance of your product being known to a model directly. Disallowing them is a legitimate position with no documented search cost in Google's case.

What you should not do is apply one blanket rule to all of them, because that decides the citation question by accident while you think you are deciding the training question.

Checking what you currently have

Three steps, and the whole thing takes ten minutes.

  • Open yourdomain.com/robots.txt in a browser and read it with your own eyes. If the file does not exist, nothing is blocked, which is fine.
  • Look for any Disallow: / rule and check which user agent it sits under. A Disallow: / under User-agent: * blocks everybody, including Google.
  • Check whether the agent names present are the current ones. A file naming Claude-Web or Anthropic-AI was written against deprecated names and is not doing what its author intended.
  • If your platform generates robots.txt for you, confirm what it generated rather than assuming, because some templates ship with restrictive defaults.

The wider point

It is worth stepping back from the file for a moment. The reason this matters more for a new product than for an established one is the asymmetry described elsewhere in this section: the AI channel is where a young domain is least disadvantaged.

Which means a blanket block costs a new product proportionally more than it costs an established one. The site with ten years of history and strong rankings loses a supplementary channel. The product launched last month loses the channel that was most likely to work first.

Read the file today. It is the shortest task in this entire section and it is the one that can invalidate all the others.

Part of a larger guide

This page is one part of The AI channel. The other parts:

Questions people ask

What is the difference between GPTBot and OAI-SearchBot?
OpenAI documents GPTBot as the crawler for content that may be used to train its foundation models, and OAI-SearchBot separately as the agent used to surface websites in ChatGPT's search features. Blocking GPTBot is a training opt-out. Blocking OAI-SearchBot removes you from the answers, which is usually not what the site owner wanted.
Should I block AI crawlers on my website?
Not as a single group. Allow the search and user-triggered agents so you remain citable, then make a separate deliberate decision about the training agents, which is a real choice with reasonable answers either way. One blanket rule decides the citation question by accident.
Does blocking Google-Extended hurt my search rankings?
No. Google's documentation states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. It controls whether crawled content may be used to train and ground Gemini models, which makes it a clean training opt-out separate from findability.
Does blocking PerplexityBot protect my content from training?
According to Perplexity's documentation there is nothing to protect against on that front, since it states that neither PerplexityBot nor Perplexity-User is used to crawl content for training foundation models. Blocking them removes you from a surface where new products tend to do comparatively well.
My robots.txt names Claude-Web. Is that right?
It is out of date. Claude-Web and Anthropic-AI are deprecated names. The current agents are ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for user-directed retrieval, and they can be controlled independently.
How do I check what my robots.txt is doing?
Open yourdomain.com/robots.txt in a browser and read it. Look for any Disallow rule and check which user agent it applies to, since a Disallow under User-agent asterisk blocks everybody including Google. If your platform generates the file, confirm what it generated rather than assuming.

Check your crawler access in sixty seconds

The free Get Found Check reads your robots.txt and reports which AI crawlers are currently allowed, which are blocked, and what that costs you.

Run the free check Have Licheo do it for you