A flooring company can write the best pages in its city and still never appear in a ChatGPT search answer, because of one line in one small text file. The check takes five minutes, needs no tools, and any owner can do it from a phone.
Why does one file matter?
ChatGPT reads websites for its search answers with a program that OpenAI calls OAI-SearchBot. OpenAI's own documentation says that OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features", and then, in one sentence: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers."
A site opts out in a file called robots.txt. Every website can have one, at the same place: the website address followed by /robots.txt. The file tells each reading program, by name, which parts of the site it may visit. Google uses the same file for its own reader, Googlebot, and lists "ensuring that crawling is allowed in robots.txt" among the basics for its AI answers.
How do you read the file?
Open a browser and type your address followed by /robots.txt, for example yourflooring.com/robots.txt. You will see a few lines of plain text, or a page that says the file does not exist. No file means nothing is blocked.
The lines come in groups. Each group starts with a User-agent line, which names a program, and continues with Disallow or Allow lines, which say which parts of the site that program may visit.
| What you see | What it means |
|---|---|
User-agent: OAI-SearchBot then Disallow: / | ChatGPT's search reader is blocked from the whole site |
User-agent: * then Disallow: / | Every program is blocked, including Google and ChatGPT, unless a group names it separately |
User-agent: GPTBot then Disallow: / | OpenAI's training program is blocked; ChatGPT search is unaffected |
User-agent: * then Disallow: /wp-admin/ | Only the admin area is blocked; this is normal |
Disallow: with nothing after it | Nothing is blocked |
The second row is the dangerous one. A site that was blocked while it was being built can keep this line by mistake after launch, and it hides the site from Google as well.
What do the three OpenAI programs do?
OpenAI names three programs in its documentation, and each has a separate setting.
- OAI-SearchBot reads websites for ChatGPT's search answers. This is the one a flooring company should allow.
- GPTBot is used, in OpenAI's words, "to make our generative AI foundation models more useful and safe". A company may block it if it prefers; it does not affect ChatGPT search.
- ChatGPT-User visits a page when a person asks ChatGPT to open it. OpenAI writes that it "is not used for crawling the web in an automatic fashion."
OpenAI says that "each setting is independent of the others", and that a change can take "~24 hours" to apply.
What if Cloudflare protects your site?
Cloudflare is a service that sits in front of a website to speed it up and protect it from attacks. It has its own settings for AI programs, separate from robots.txt.
Cloudflare's documentation, updated on November 2, 2026, calls the setting "AI Crawler and Agent access policies". It groups AI programs by what they do, and defines Search as "crawlers that collect or index your content to answer questions about it later." For new domains, it says, "Search, Agents, and Training use Allow (do not block) by default."
A site set up earlier, or by someone who changed the setting, may differ. If your site uses Cloudflare, ask whoever manages it to confirm that the Search policy is set to Allow. Other security and hosting services can have similar settings, so ask the same question of yours.
What should the file say?
For most flooring companies a short file is enough. The example below is for a WordPress site, whose admin area sits at /wp-admin/; other website builders use other addresses. It allows every program, keeps the admin area private, and tells programs where the list of pages is. If you want to keep your pages out of AI training, add the GPTBot group; ChatGPT search remains allowed.
User-agent: *
Disallow: /wp-admin/
User-agent: GPTBot
Disallow: /
Sitemap: https://www.yourflooring.com/sitemap.xml
Do not edit the file yourself unless you manage the site. Send the lines you found to whoever does, and ask them to allow OAI-SearchBot and Googlebot. Check the file again the next day.
Is allowing the program enough?
Allowing the program makes the site eligible to be shown. What the assistant then writes depends on what it finds on the site and about it.
We saw this in our test of Gemini on November 8, 2026: 20 questions about flooring companies in 10 cities (the full test). The answers were built mostly from the companies' own websites, 281 of 390 statements, and the facts repeated were plain ones: the year started, the towns served, the floors installed, the crews, the warranty, the estimate and the steps of the job. We have not yet run the same test on ChatGPT, so we report only what OpenAI documents.
The next steps are in our flooring company marketing hub, starting with how ChatGPT and Gemini choose a flooring company and what a flooring website needs.
At Licheo we build and run websites for small businesses so they get found on Google and in AI answers; contact us for pricing. The free Get Found Check reads your homepage and returns a letter grade, with a separate grade for readiness for AI assistants, in under 60 seconds.
Sources
- Overview of OpenAI Crawlers, OpenAI, accessed November 8, 2026.
- AI features and your website, Google Search Central, last updated 2025-12-10, accessed November 8, 2026.
- AI Crawler and Agent access policies, Cloudflare, last updated November 2, 2026, accessed November 8, 2026.
- Licheo test of Gemini 2.5 Pro with grounding in Google Search: 20 questions about flooring companies in 10 cities, run on November 8, 2026.