Skip to main content
SEO

robots.txt

robots.txt is a text file placed at the root of a website that gives crawling instructions to bots, specifying which paths they may or may not request.

robots.txt is a request, not an enforcement mechanism. Well-behaved bots (Googlebot, Bingbot, Anthropic's ClaudeBot, OpenAI's GPTBot) respect it; malicious bots ignore it. Never rely on robots.txt for security — it is a public file that effectively advertises what you do not want crawled.

Common patterns: `Disallow: /admin/` keeps internal pages out of search results, `Disallow: /api/` keeps API endpoints from being crawled, `Allow: /` (the default) opens everything else. The `Sitemap:` directive points crawlers to your sitemap.

A frequent mistake is `Disallow: /` shipped to production. This blocks all crawling and silently removes the site from search results. Always check robots.txt after a deploy, and always validate in Search Console.

Related terms

Keep exploring

Related resources

Browse the full glossary

Explore every term across email marketing, SEO, ecommerce, content marketing, and social media.

Newsletter

Stay Ahead

Occasional, engineering-led notes on applied AI — what we're building, running, and learning in production.