Skip to main content
Webanto LogoWebanto Logo
AboutBlogPortfolioProductsServices
  1. Home
  2. Glossary
  3. Robots Txt
Newsletter

Stay in Orbit

Get the latest digital insights delivered to your inbox — strategies, trends, and tips from the frontier of web and marketing.

Webanto LogoWebanto Logo

Out of This World Digital Solutions. We help businesses reach new heights with cutting-edge technology and innovative strategies.

Services
Website DevelopmentSEO OptimizationSocial MediaContent Marketing
Company
About UsPortfolioLatest NewsDocumentationContactClient Portal
Launch Your Mission

Ready to launch your project into the digital stratosphere? Let's build something extraordinary.

Start Project
© 2026 Webanto. Engineered for the future.
TermsPrivacyContact
SEO

robots.txt

robots.txt is a text file placed at the root of a website that gives crawling instructions to bots, specifying which paths they may or may not request.

robots.txt is a request, not an enforcement mechanism. Well-behaved bots (Googlebot, Bingbot, Anthropic's ClaudeBot, OpenAI's GPTBot) respect it; malicious bots ignore it. Never rely on robots.txt for security — it is a public file that effectively advertises what you do not want crawled.

Common patterns: `Disallow: /admin/` keeps internal pages out of search results, `Disallow: /api/` keeps API endpoints from being crawled, `Allow: /` (the default) opens everything else. The `Sitemap:` directive points crawlers to your sitemap.

A frequent mistake is `Disallow: /` shipped to production. This blocks all crawling and silently removes the site from search results. Always check robots.txt after a deploy, and always validate in Search Console.

Related terms

  • XML Sitemap

    An XML sitemap is a file (typically `sitemap.xml`) that lists the URLs on your site you want search engines to crawl and index, along with metadata like last-modified date and update frequency.

  • Canonical URL

    A canonical URL is the preferred version of a page, declared via the `<link rel='canonical'>` tag, that tells search engines which URL to index when the same content is reachable at multiple URLs.

← Orphaned PageSchema Markup →

Browse the full glossary

Explore every term across email marketing, SEO, ecommerce, content marketing, and social media.