Robots.txt
A site-root text file that communicates crawler access rules under the Robots Exclusion Protocol.
A correct robots.txt helps manage crawler access and protect crawl capacity from low-value paths while pointing crawlers to sitemaps.
A site allows public pages, disallows an internal API path, and declares the canonical XML sitemap while keeping CSS and JavaScript required for rendering accessible.
Robots.txt is not an access-control or confidentiality mechanism and does not itself prevent indexing. Blocked URLs may still appear without content, and sensitive resources require authentication or removal.
Rules identify user agents and paths that are allowed or disallowed for crawling and may declare sitemap locations. The file is publicly accessible, path matching is sensitive to syntax, and support for directives varies among crawlers.
Systems Architecture
Google Search Central — robots.txt — https://developers.google.com/search/docs/crawling-indexing/robots/intro; RFC 9309 — Robots Exclusion Protocol — https://www.rfc-editor.org/rfc/rfc9309