Crawl Budget
The number of URLs a search engine can and wants to crawl on a site in a given period, set by server capacity and crawl demand.
On large catalogs, marketplaces and news sites, wasted crawling of duplicate or parameter URLs can delay discovery of new and updated pages, which postpones the traffic and revenue those pages would bring.
An online retailer with 1.8 million URLs finds that faceted filter combinations take most Googlebot requests in the Crawl Stats report, blocks those parameters in robots.txt, fixes 404 links and updates lastmod values in its XML Sitemap.
Google's guidance targets sites with over 10,000 pages that change daily or over 1 million pages; smaller sites usually need only a current sitemap. Noindex does not save crawl budget, because the page must still be fetched.
Google defines the crawl capacity limit as the connections and time Googlebot can use without overloading a server, rising with fast responses and falling with 5xx or 429 errors. Crawl demand reflects perceived inventory, popularity and staleness. Efficiency improves by consolidating duplicates, blocking low-value URLs in Robots.txt, returning correct status codes, avoiding redirect chains, supporting 304 responses and keeping the XML Sitemap current, alongside fixing Crawlability and Orphan Page issues.
Systems Architecture
Google Search Central — Crawl budget management for large sites — https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget; Search Console Help — Crawl Stats report — https://support.google.com/webmasters/answer/9679690