Digital Lexicon Hub

The review of raw web server access logs to see how search engine crawlers and other bots actually request a site's URLs.

Logs are the only first-party record of what crawlers really fetch, so they show whether search engines spend their visits on revenue pages or on parameters, redirects and errors that never earn traffic.

An SEO team exports 30 days of Nginx logs, keeps hits whose user agent claims Googlebot, verifies them by reverse DNS, finds 38% of requests on faceted filter URLs, then blocks those parameters in robots.txt and tracks the shift.

Logs show requests, not rankings or indexing decisions, and CDNs or load balancers may hold the real records. Fake Googlebot user agents are common, and logs contain IP addresses that count as personal data under GDPR.

Each access log line records fields such as client IP, timestamp, requested URL, HTTP status code, bytes sent, referrer and user agent. Analysts parse these lines, keep only verified search engine bots (Google recommends reverse and forward DNS lookups or its published IP ranges), then group hits by URL pattern, status code and date. The result shows how Crawl Budget is spent, which sections are never crawled, and where Crawlability problems such as 5xx errors or redirect loops absorb requests.