SEO & GEO glossary

robots.txt

robots.txt is a plain-text file at the root of a website that tells crawlers which parts of the site they may or may not crawl. Its rules follow the Robots Exclusion Protocol, standardised as RFC 9309 in 2022.

Also known as: Robots.txt file, Robots Exclusion Protocol

Why it matters

Used well, robots.txt keeps crawlers away from endless filtered URLs, internal search results and admin areas, so they spend their time on pages that matter. Used carelessly, one line can block an entire site, or block the CSS and JavaScript files a search engine needs to render pages the way visitors see them.

It controls crawling, not indexing. Google’s introduction to robots.txt (opens in a new tab) warns it is not a way to keep a page out of search results: a blocked URL can still be indexed if other pages link to it, so use a noindex rule for that. The same file is where you allow or block AI crawlers, such as PerplexityBot or Common Crawl’s CCBot.

Example

A shop adds the lines “User-agent: *” and “Disallow: /search/” so crawlers skip its internal search result pages, plus “Sitemap: https://example.com/sitemap.xml” so they can find the pages it does want crawled.

Sources

Put the theory to work on your own site

DraftSEO is an AI SEO writer that researches keywords, writes SEO- and GEO-optimised articles with images, internal links and FAQs, and auto-publishes them to your website on a schedule.

Start for free