Digital Lexicon Hub

A robots directive requesting that a search engine exclude a page or resource from its searchable index.

Noindex keeps low-value, duplicate or private pages out of search results, which focuses search visibility on pages that matter and prevents thin content such as internal search results from appearing.

An e-commerce site adds to 1,800 filtered category URLs and internal search pages, keeps them crawlable in robots.txt, and tracks their removal in the Search Console Page indexing report.

If robots.txt blocks a page, crawlers cannot see the noindex rule and the URL may stay indexed. Noindex does not restrict access, so it is not a security control, and long-term noindexed pages may be crawled less often.

Noindex is set either with a robots meta tag in the HTML head, such as meta name="robots" content="noindex", or with an X-Robots-Tag HTTP response header, which also works for PDFs and images. The rule takes effect only after the crawler fetches the URL and reads it. A crawler-specific name such as googlebot can replace robots to target one search engine.