llms.txt: Docs for Machines
Context for models, not for ranking: why llms.txt and llms-full.txt help coding agents – and what an MCP server makes of them.
llms.txt sits in the root directory of a website, which makes it look like the next signal for search engines – a neighbor of robots.txt and sitemap.xml, created in the hope of visibility. Exactly this conclusion misses the point. The file was never meant as a ranking signal, and there is no evidence of any ranking effect. Its value doesn’t lie in crawler signals but in providing agents with curated documentation context.
There are two files, llms.txt and llms-full.txt. They look like marketing infrastructure, and their value lies somewhere else: in the question of how a model gets clean, curated context without first working its way through navigation bars, cookie banners and JavaScript. Both files are machine-readable self-documentation of a product.
Two files, one problem#
Jeremy Howard, co-founder of Answer.AI, published the proposal on September 3, 2024; the description is at llmstxt.org. The starting point is a sober observation: models increasingly rely on web content, but their context window is too small to take in an entire website in one go. And turning raw HTML into usable text is tedious and imprecise – between the actual content there is a lot of clutter that only eats up space.
The answer consists of two building blocks that you should keep clearly apart.
llms.txt sits in the root directory of a site and is a curated index in Markdown. The format is deliberately narrow: an H1 heading with the project name, below it a blockquote with a short summary, optionally a few sentences of context, then H2 sections with lists of links. Each link carries a brief description. A separate ## Optional section marks what a model may drop first when space is tight. This is how compact it looks:
# Orderflow API
> Orderflow is a REST API for creating, tracking, and refunding orders.
> These docs are written for LLMs and coding agents.
Every write endpoint accepts an `Idempotency-Key` header, so retries stay safe.
## Core Documentation
- [Quick Start](https://docs.example.com/quickstart.md): Authenticate and create your first order
- [API Reference](https://docs.example.com/api.md): Endpoints, parameters, and error codes
- [Webhooks](https://docs.example.com/webhooks.md): Event types and signature verification
## Optional
- [Changelog](https://docs.example.com/changelog.md): Version history and breaking changes
llms-full.txt is the second building block and conceptually simpler: the full text of the important pages, concatenated into a single Markdown file. No longer an index, but the whole content in one piece, without HTML noise. An agent or client can load the bundled content with a single request instead of working its way through page by page – whether all of it fits the specific task is another question.
In addition, Howard suggests also providing individual pages as a clean Markdown version at the same address with .md appended – .../api becomes .../api.md. That’s the glue between index and content: the index points to pages that already exist in the form a model processes best anyway.
The distinction matters, because a lot gets mixed up here. robots.txt governs what a crawler may touch. sitemap.xml helps with indexing. llms.txt does neither: the file puts things in context and curates them, intended for the moment of inference, not for the crawl. An agent that reads it gets an editorial statement about what matters in a product – not a promise to a search engine. The three files don’t exclude each other; they answer different questions.
flowchart TD
A[Coding agent] -->|knows llms.txt| B[llms.txt<br/>curated index]
B --> C{Which page<br/>do I need?}
C -->|targeted| D[page.md<br/>clean Markdown version]
C -->|everything at once| E[llms-full.txt<br/>entire content]
A -.->|alternative without llms.txt| F[Crawl the HTML page<br/>navigation, ads, JS]
F -.->|lots of noise| G[Context window full,<br/>content diluted]Why this helps#
When integrating a third-party library, an agent without a Markdown view often loads the complete HTML page or improvises missing signatures. In the first case, it pulls in the menu, the footer and three code examples it doesn’t need at all – a noticeable part of the window is gone before the actual task begins. In the second case, it guesses. In a code review, I noticed that an agent had simply made up an API signature, plausible and wrong, because fetching the exact reference was too expensive for it. Both are symptoms of the same problem: too much noise per token, too little reliable substance.
llms.txt addresses exactly this. Curated Markdown without layout ballast means more of the window is left for the actual content. The index also forces a decision: not every page is equally important, and someone with domain knowledge has to name which thirty to a hundred pages make up the core. And when the model needs the exact error message or the exact parameter, it fetches the matching .md page instead of improvising.
The point about curation is often overlooked. A good llms.txt isn’t an arbitrary list of links but an editorial statement about what matters in a product. This decision is the actual work – not the file itself.
Where it pays off#
For a blog or a marketing site, llms.txt brings little: there is no evidence of a ranking effect, and if you create the file hoping for better visibility, you’ll be disappointed. Anyone who promises such an effect promises too much.
The exception is narrow but real: technical documentation and coding agents. An API reference, a component library, an SDK – content that is maintained in Markdown anyway and that an agent in the IDE looks up in a targeted way.
For pure marketing pages without demonstrable agent usage, the maintenance effort usually doesn’t justify the file; for regularly queried technical documentation, it can pay off. Server logs with requests to /llms.txt give you a hint of usage, but they don’t carry the decision on their own: a lack of direct requests doesn’t necessarily prove a lack of usage, for example when the same content is retrieved through other paths such as MCP resources.
Generated from the source, not maintained by hand#
A central risk when using these files is organizational, not technical: two versions of the same content that are maintained separately. The page for humans here, the Markdown version for models there. If llms-full.txt lags behind the real docs, you feed models outdated content – and that’s worse than no file at all, because it creates false trust. Without automatic generation from the same source, drift is only a matter of time.
That’s why both files belong in the build, not in the editor. The build can generate llms.txt, llms-full.txt and Markdown twins per site and locale from the published state. What matters is the shared source for human-readable and machine-readable outputs; a second, hand-maintained documentation would make drift possible again.
That’s the second half of the curation work. A human makes the editorial decision about what belongs in the index. The build takes care of generating the files. If you do both by hand, you have taken on a second set of docs that easily deviates from the published state.
What an MCP server makes of it#
Up to this point, llms.txt is a static file that someone picks up at some point. MCP adds targeted access to resources and operations on top of the static index. The Model Context Protocol is the open protocol that Anthropic introduced at the end of 2024, through which an agent talks to external tools and data sources in a standardized way.
As an index, llms.txt describes which content exists and where it lives; an MCP server provides resources and operations in a targeted way. With mcpdoc, the LangChain team built exactly such a docs server. It receives a list of llms.txt addresses and provides the agent with two tools: one that lists the registered sources and one that fetches a specific page from them.
An MCP docs server designed for this can load context in a targeted way, make calls loggable and restrict access – three improvements over simply downloading the file.
First, dosage. Instead of dumping all of llms-full.txt into the window, the agent uses the index to request only the one page that fits the task – the range between “no context at all” and “everything at once.”
Second, traceability. With a docs server like mcpdoc, fetching runs through tool calls that you can inspect: which address was loaded and what came back. This doesn’t automatically apply to resource reads; there, you need your own logging on the server side. A direct download of the file doesn’t necessarily leave this trail in the agent workflow. For internal or protected documentation, this auditability can justify the additional operational effort.
Third, access control. You can restrict the server to allowed domains, and for internal or protected docs you put authentication in front of it. If you need permission-bound access, you can offer the documentation as a resource through an authenticated MCP server and check permissions there for every read. This way, the convention also works behind the company firewall: the server points to a private llms.txt, and models in the team get curated knowledge without the content being openly available on the web.
If both come from one product, it can offer a remote MCP interface alongside the generated files. In such a setup, MCP resources are application-managed context that the server offers the client as documentation. Tools stay separate from this: they represent domain operations and should only become visible with the matching scope. Such a system can additionally provide its own llms.txt documentation as an MCP resource, so that a client can read the tool model and API boundaries without guessing a web page. So MCP adds targeted access and operations to the static documentation; this access only becomes permission-bound once the server authenticates and authorizes resource reads. The documentation doesn’t replace MCP – without generated, up-to-date machine views, these documentation resources would lack a reliable foundation.
flowchart LR
H[Agent host<br/>Cursor, Claude Code] --> M[MCP docs server<br/>e.g. mcpdoc]
M -->|list sources| L[llms.txt<br/>registered indexes]
M -->|fetch page| P[only the needed<br/>.md page]
P --> Hllms-full.txt in particular gains from this combination. You can attach the file directly to an MCP server as a knowledge source – without an embedding pipeline, without a vector database. More accurate than “a replacement for RAG” is to describe it as a tool-based, auditable variant of it: not unpredictable retrieval from a large vector index, but traceable access to curated pages.
Where it falls short#
llms.txt is a proposal, not an adopted standard. There is no committee behind it and no guarantee that a particular model reads the file. You provide it; you can’t force it to be consumed.
Size is a real limit. An llms-full.txt that is several megabytes in size may, depending on tokenization, model and client limits, not fit completely into the available context and then only gets loaded in fragments – and with that, the advantage of “everything in one request” is gone. For large sites, you stick with the curated index of the truly central pages, complemented by targeted fetching of individual .md versions.
And finally, the question of trust. As soon as an agent takes over a third-party llms-full.txt without checking it, it takes over content it doesn’t control. That’s the same attack surface as with any external data source that ends up in the context – a good reason to go through an MCP server with a domain restriction instead of collecting arbitrary addresses.
Conclusion#
llms.txt and llms-full.txt are machine-readable self-documentation. The index says what matters and where it lives; the full version delivers the content in one piece; the .md mirrors provide individual pages in the form models prefer. None of them is a reliable ranking factor, and if you create the files with that expectation, you’re measuring against the wrong goal. Their value arises where documentation meets agents – with SDKs, API references and coding tools.
Two decisions follow from this. First: the files are generated from the same source as the documentation for humans, or you leave them out – a hand-maintained duplicate drifts and undermines exactly the trust it’s supposed to build. Second: if you need targeted or permission-bound access, you can offer the documentation as a resource through an authenticated MCP server and check permissions there. In this way, MCP can complement the static views with measured, auditable and – with server-side authorization – permission-bound access; it still depends on the documentation behind it being correct.
Further reading#
- The original proposal and the format description: llmstxt.org
- Repository for the proposal: AnswerDotAI/llms-txt
- MCP docs server that reads
llms.txtas a source: langchain-ai/mcpdoc - Model Context Protocol: modelcontextprotocol.io
Revised and translated from German with the help of AI.