The Robots Exclusion Protocol, commonly known as robots.txt, is one of the oldest standards on the web, originally proposed by Martijn Koster in 1994 and formalized as an IETF standard in RFC 9309 (published in 2022, nearly three decades after its informal adoption). The protocol provides a mechanism for website owners to communicate with web crawlers about which parts of their site should or should not be accessed. It operates on a voluntary compliance model: well-behaved crawlers honor robots.txt directives, but malicious bots or scrapers may ignore them entirely.

The robots.txt file must be placed at the root of a website (https://example.com/robots.txt) and uses a simple text-based syntax. Each section begins with a User-agent directive specifying which crawler the rules apply to (or * for all crawlers), followed by Allow and Disallow directives that specify URL paths the crawler may or may not access. The Crawl-delay directive, though not part of the original standard, is honored by some crawlers (notably Bing and Yandex) to limit request frequency. The Sitemap directive points crawlers to XML sitemap files that list all pages the site wants indexed, complementing the exclusion rules with inclusion guidance.

The relationship between robots.txt and SEO is nuanced and frequently misunderstood. Blocking a URL with Disallow prevents crawlers from accessing the page, but it does not prevent the URL from appearing in search results. If other pages link to a disallowed URL, search engines may still index the URL with limited information, displaying the link text from referring pages. To prevent a page from appearing in search results entirely, the page must return a noindex meta robots tag or X-Robots-Tag HTTP header, which requires the page to be crawlable. This means that using robots.txt to block sensitive pages can paradoxically make them more visible in search results by preventing the search engine from seeing the noindex directive.

The emergence of AI training bots has created a new dimension for robots.txt usage. Crawlers like GPTBot (OpenAI), ChatGPT-User (OpenAI), Google-Extended (Google), anthropic-ai (Anthropic), and CCBot (Common Crawl) scrape web content for training large language models. Many website owners now add specific Disallow rules for these user agents to prevent their content from being used in AI model training. This has reignited debate about the adequacy of the robots.txt standard, which was designed for search engine crawling and lacks granularity for distinguishing between different uses of crawled content such as indexing, caching, and AI training.