Digital Lexicon Hub logo

Digital Lexicon Hub

THE TECHNICAL LANGUAGE OF DIGITAL B2B INFRASTRUCTURE

CONCEPT
SEARCH OPTIMIZATION

Crawl Budget

The number of URLs a search engine can and wants to crawl on a site in a given period, set by server capacity and crawl demand.


BUSINESS RELEVANCE

On large catalogs, marketplaces and news sites, wasted crawling of duplicate or parameter URLs can delay discovery of new and updated pages, which postpones the traffic and revenue those pages would bring.


IMPLEMENTATION EXAMPLE

An online retailer with 1.8 million URLs finds that faceted filter combinations take most Googlebot requests in the Crawl Stats report, blocks those parameters in robots.txt, fixes 404 links and updates lastmod values in its XML Sitemap.


LIMITATIONS

Google's guidance targets sites with 1 million+ pages that change weekly, 10,000+ pages that change daily, or many URLs left as Discovered - currently not indexed; other sites usually need only a current sitemap. Noindex does not save crawl budget, because the page must still be fetched.


TECHNICAL EXPLANATION

Google defines the crawl capacity limit as the connections and time Googlebot can use without overloading a server, rising with fast responses and falling with 5xx or 429 errors. Crawl demand reflects perceived inventory, popularity and staleness. Efficiency improves by consolidating duplicates, blocking low-value URLs in Robots.txt, returning correct status codes, avoiding redirect chains, supporting 304 responses and keeping the XML Sitemap current, alongside fixing Crawlability and Orphan Page issues.


Secondary Topics

Systems Architecture

Sources

Google Crawling Infrastructure — Crawl Budget Management — https://developers.google.com/crawling/docs/crawl-budget; Search Console Help — Crawl Stats report — https://support.google.com/webmasters/answer/9679690

Related terms

Lookalike Audience

CONCEPT
PERFORMANCE ADVERTISING

Lookalike audiences let Meta, Google and other ad platforms find new users who resemble a seed list of customers, site visitors or high-value buyers.

Model Card

CONCEPT
AI GOVERNANCE

A standardized technical document outlining the performance benchmarks, limitations, and intended use cases of an AI model.

Microcopy

CONCEPT
EXPERIENCE DESIGN

Microcopy covers the small pieces of text in an interface, from labels to error messages, that tell users what to do and reassure them at decision points.

Data Layer

CONCEPT
DATA ENGINEERING

How a website data layer exposes structured page, product and event information so tag managers can fire tags without scraping the page.

Idempotency

CONCEPT
SYSTEMS ARCHITECTURE

A software design property ensuring that repeating an operation, such as an API request, has the same intended effect as running it once.