Orphan Page
CONCEPT
SEARCH OPTIMIZATION
A website page lacking internal inbound links from any other page on the domain, limiting discoverability and crawling.
CONCEPT
SEARCH OPTIMIZATION
Robots.txt
A site-root text file that communicates crawler access rules under the Robots Exclusion Protocol.
BUSINESS RELEVANCE
A correct robots.txt helps manage crawler access and protect crawl capacity from low-value paths while pointing crawlers to sitemaps.
IMPLEMENTATION EXAMPLE
A site allows public pages, disallows an internal API path, and declares the canonical XML sitemap while keeping CSS and JavaScript required for rendering accessible.
LIMITATIONS
Robots.txt is not an access-control or confidentiality mechanism and does not itself prevent indexing. Blocked URLs may still appear without content, and sensitive resources require authentication or removal.
TECHNICAL EXPLANATION
Rules identify user agents and paths that are allowed or disallowed for crawling and may declare sitemap locations. The file is publicly accessible, path matching is sensitive to syntax, and support for directives varies among crawlers.
Secondary Topics
Systems Architecture
Sources
Google Search Central — robots.txt — https://developers.google.com/search/docs/crawling-indexing/robots/intro; RFC 9309 — Robots Exclusion Protocol — https://www.rfc-editor.org/rfc/rfc9309
Related terms
Out-of-Home Advertising (OOH)
CONCEPT
PERFORMANCE ADVERTISING
Outdoor advertising media targeting consumers in public transit hubs, commercial displays, and roadside billboards.
Search Experience Optimization (SXO)
CONCEPT
SEARCH OPTIMIZATION
A holistic discipline combining search intent optimization with conversion-focused user experience design.
Design System
CONCEPT
EXPERIENCE DESIGN
A unified collection of repeatable UI components, guidelines, tokens, and design standards ensuring digital product consistency.
301 Redirect
CONCEPT
SEARCH OPTIMIZATION
Permanent HTTP 301 redirects preserve search equity and guide user journeys during URL changes, migrations, and site restructuring.