Crawlability
CONCEPT
SEARCH OPTIMIZATION
The degree of ease and efficiency with which search engine spiders discover, access, and traverse website pages and content.
CONCEPT
SEARCH OPTIMIZATION
XML Sitemap
A machine-readable file that lists canonical URLs and optional metadata to help search engines discover and monitor a website’s indexable resources.
BUSINESS RELEVANCE
Accurate sitemaps improve discovery and monitoring for large, new, frequently changing, or weakly linked sites and provide useful coverage signals in search-engine tools.
IMPLEMENTATION EXAMPLE
A multilingual site generates separate sitemap files containing only canonical, indexable 200-status URLs, adds accurate lastmod dates, groups them in a sitemap index, and declares it in robots.txt.
LIMITATIONS
A sitemap is a discovery hint, not a ranking or indexing guarantee. Including redirected, duplicate, blocked, non-canonical, or stale URLs creates conflicting signals, and lastmod should not be changed without meaningful updates.
TECHNICAL EXPLANATION
The Sitemap protocol expresses URL locations in an XML urlset or references multiple files through a sitemap index. Optional lastmod values should reflect significant content changes; specialised extensions can describe images, video, or news where supported. Sitemaps may be declared in robots.txt and submitted to search engines.
Secondary Topics
Systems Architecture
Sources
http://Sitemaps.org — Protocol — https://www.sitemaps.org/protocol.html; Google Search Central — Build and submit a sitemap — https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
Related terms
Brand Safety
CONCEPT
PERFORMANCE ADVERTISING
Strategic measures and filtering controls designed to prevent digital ads from appearing next to inappropriate or harmful content.
Hreflang
CONCEPT
SEARCH OPTIMIZATION
An HTML attribute signaling language and regional page variations to search engines for localized search results delivery.
Exception Handling
CONCEPT
INTELLIGENT AUTOMATION
Software routines that detect, manage, isolate, and recover from runtime execution errors without collapsing operations.
Robots.txt
CONCEPT
SEARCH OPTIMIZATION
A standard text configuration file situated at the website root instructing web crawlers which directory paths to avoid.