llms.txt
A proposed Markdown file placed on a website that gives large language models a curated summary of the site and links to its most useful content.
As AI assistants answer more questions directly, llms.txt gives a business a low-cost way to state which product, pricing and documentation pages best represent it, although AI providers are not obliged to read it.
A SaaS company publishes /llms.txt with an H1 naming the product, a one-line blockquote summary, H2 lists linking to 12 documentation pages and its pricing page, an Optional section for blog archives, and .md copies of each linked page.
llms.txt is a community proposal, not a ratified standard, and each AI provider decides whether to use it. It does not control crawler access or training use as Robots.txt rules aim to, and an outdated file can point models to stale information.
The proposal, published by Jeremy Howard of Answer.AI in September 2024, defines a Markdown file whose only required element is an H1 with the site or project name. Optional parts follow in a fixed order: a blockquote summary, free-text detail, H2 file lists of named links with notes, and an Optional section that agents may skip. Sites can also serve Markdown copies of pages at the same URL with .md appended. It complements Robots.txt and the XML Sitemap rather than replacing them.
Intelligent Automation, Systems Architecture
llms-txt (Answer.AI) — The /llms.txt file — https://llmstxt.org/; Answer.AI — /llms.txt: a proposal to provide information to help LLMs use websites — https://www.answer.ai/posts/2024-09-03-llmstxt.html