Category: Technical SEO
A text file at the root of a website that tells search engine crawlers which pages or sections they are allowed or not allowed to access. It is the first file crawlers check before exploring a website.
Robots.txt is a plain text file placed at the root of a website (accessible at yoursite.com/robots.txt) that provides instructions to search engine crawlers about which parts of the site they should and should not access. It follows the Robots Exclusion Protocol, a standard that has been in use since the earliest days of the web, and it remains one of the first things any search engine bot checks before it begins crawling your site.
The syntax of robots.txt is deceptively simple — User-agent directives specify which crawlers the rules apply to, Disallow directives block access to specific paths, and Allow directives permit access within otherwise blocked areas. Yet this simplicity masks significant power and risk. A single misconfigured line in robots.txt can block search engines from crawling your entire website, effectively making you invisible in search results. It is not uncommon for a well-intentioned developer to add a Disallow directive during staging and forget to remove it before launch — with devastating consequences.
It is essential to understand what robots.txt does and does not do. It instructs well-behaved crawlers not to access certain pages, but it does not prevent indexing if search engines discover those URLs through other means (such as links from other websites). If you truly need to prevent a page from appearing in search results, you should use a noindex meta tag or X-Robots-Tag header instead. Robots.txt is a crawling directive, not an indexing directive — a distinction that causes considerable confusion.
For most websites, the robots.txt file should be relatively simple: allow crawling of all important content, block admin areas and internal search results that add no SEO value, and include a reference to your XML sitemap. Regularly audit your robots.txt file as part of your technical SEO maintenance — particularly after website migrations, redesigns, or CMS changes. The consequences of blocking important content are too severe to leave this file unmonitored.
Understanding SEO terminology is the first step. The next step is knowing where your website stands. Run a free AI-powered SEO audit with Licheo and discover exactly what needs attention.