The file that can silently switch off your whole site

It is a plain text file with a handful of lines, it sits at one fixed address, and a single character in the wrong place makes everything else on your site invisible without anything appearing to be wrong.

Run the free check

robots.txt tells crawlers which parts of your site they may request. The dangerous line is Disallow followed by a single slash under User-agent asterisk, which blocks everybody from everything. In 2026 the file also has to distinguish between training crawlers and search crawlers, because blocking them as one group removes you from AI answers while you believe you are opting out of training.

What the file is

robots.txt is a plain text file that lives at the root of your domain, at yourdomain.com/robots.txt, and nowhere else. Crawlers request it before crawling and it tells them which paths they may fetch.

It is a set of instructions that well-behaved crawlers follow voluntarily. Google, Bing, and the major AI companies all honour it. It is not a security mechanism, so anything you genuinely need to keep private belongs behind authentication rather than behind a Disallow line.

Not having the file at all is a perfectly valid state and means nothing is restricted. An empty or missing robots.txt is much safer than a wrong one.

The line that causes the damage

Two lines, in this combination, block everything from everybody: User-agent set to an asterisk, followed by Disallow set to a single forward slash.

The asterisk means every crawler. The lone slash means every path on the site. Together they say nobody may fetch anything.

This appears in real sites constantly, for a small number of reasons. It is the default in some starter templates. It gets added deliberately while a site is in development and then never removed at launch. And it gets copied from a tutorial without the reader noticing what it does.

The reason it survives so long is that nothing looks wrong. Your site loads, your product works, your own visits appear in analytics. The only symptom is that you are never found, which looks exactly like being new.

Reading your own file right now

Open yourdomain.com/robots.txt in a browser. Three possible outcomes, and each means something different.

  • Nothing there, or a 404. Nothing is blocked. This is fine and you can move on.
  • A file with Allow lines or with Disallow lines naming specific paths such as an admin area. Read them and confirm none of them cover pages you want found.
  • A Disallow with a lone slash. Find out which User-agent it sits under. If that is an asterisk, this is your problem and it is the most valuable thing you will fix this week.
  • If your platform generates the file for you, still read it. Some templates ship with restrictive defaults and assuming is how this goes unnoticed for months.

The 2026 complication

A file that was sensible two years ago may be wrong now, because the crawlers have split into categories that did not exist then.

OpenAI documents GPTBot as the crawler for content that may be used to train foundation models, and OAI-SearchBot separately as the agent used to surface websites in ChatGPT's search features. ChatGPT-User visits pages when a user asks a question. Anthropic draws the same distinction between ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for user-directed retrieval.

A great deal of advice published recently told people to block AI bots, and the snippets that circulated named every AI agent the author could think of. Applied literally, they remove you from the answers as well as from the training, which is almost never what the site owner intended.

Perplexity is the clearest case. It documents PerplexityBot and Perplexity-User and states that neither is used to crawl content for training foundation models. Blocking them declines citations and gets nothing in return.

What a sensible file contains now

There is no single correct file, because the training question is a genuine choice on which reasonable people disagree. But the structure below expresses the position most small products actually hold.

Allow the ordinary search crawlers, Googlebot and Bingbot, with no restrictions on your public pages. Allow the AI search and user-triggered agents: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. These are what make you citable.

Then decide once, deliberately, about the training agents: GPTBot, ClaudeBot and Google-Extended. Google's documentation states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal, which makes it a clean training opt-out with no search cost.

Finally, add a line pointing at your sitemap, since robots.txt is a conventional place for crawlers to discover it.

Common mistakes beyond the obvious one

A few smaller errors that cost less but still cost something.

  • Blocking a path that assets live under, so crawlers cannot fetch your stylesheets or scripts and see a broken version of the page.
  • Naming deprecated agents. Claude-Web and Anthropic-AI are deprecated names, so a file naming only those is not doing what its author intended.
  • Putting the file anywhere other than the root. It only works at yourdomain.com/robots.txt.
  • Assuming Disallow removes a page from search results. It stops the crawl, and a page blocked this way can still appear if other sites link to it. Use a noindex directive on the page itself when you genuinely want it excluded.
  • Editing the file and never checking the live version, especially on platforms that generate it during deployment and may overwrite yours.

Part of a larger guide

This page is one part of The fixes. The other parts:

Questions people ask

What does Disallow slash do in robots.txt?
Under User-agent asterisk it blocks every crawler from every path on your site. It is the single most damaging line the file can contain, and it survives unnoticed because nothing looks broken: the site loads, the product works, and the only symptom is never being found, which looks like being new.
Do I need a robots.txt file at all?
No. A missing file means nothing is restricted, which is a valid and safe state. An empty or absent robots.txt is far better than a wrong one, so if you are unsure what yours says, deleting it is safer than guessing.
Should I block AI crawlers in robots.txt?
Not as a single group. Allow the search and user-triggered agents such as OAI-SearchBot, Claude-SearchBot and PerplexityBot so you stay citable, then decide separately about the training agents GPTBot, ClaudeBot and Google-Extended. One blanket rule decides the citation question by accident.
Does blocking Google-Extended hurt my rankings?
No. Google's documentation states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. It controls whether content may be used to train and ground Gemini models, making it a clean training opt-out.
Will Disallow remove my page from Google?
Not reliably. Disallow stops the crawl, but a blocked page can still appear in results if other sites link to it, since Google knows the URL exists without having read it. To genuinely exclude a page, use a noindex directive on the page itself and allow the crawl so it can be seen.
My platform generates robots.txt. Do I still need to check it?
Yes. Some templates ship with restrictive defaults, and some platforms regenerate the file during deployment and overwrite yours. Read the live version at yourdomain.com/robots.txt rather than assuming what it contains.

Have your robots.txt read for you

The free Get Found Check fetches your robots.txt and reports exactly which crawlers are allowed and which are blocked right now.

Run the free check Have Licheo do it for you