llms.txt is a Markdown file placed at a site's root to help LLMs and agents navigate its content. It blocks no crawlers, guarantees no citations, and Google has confirmed it doesn't use it. Publishing one costs an hour and can't hurt — but it is not an AI visibility lever, and this glossary entry explains why.
- What it is: a Markdown table of contents for agents, proposed by Jeremy Howard (Answer.AI) in September 2024.
- What it does: helps LLMs navigate at inference time — real adoption sits mostly in technical docs.
- What it doesn't do: block crawlers (that's robots.txt's job) or guarantee citations (no public evidence).
- Google's position: "no AI system uses llms.txt" (John Mueller, 2025).
- Verdict: an asymmetric bet — publish it, but expect nothing on the citation side.
What is an llms.txt file?
llms.txt is a proposed standard published on September 3, 2024 by Jeremy Howard, co-founder of Answer.AI. The idea fits in one sentence: a Markdown file served at the site root (/llms.txt) that describes the site and lists its key pages, so an LLM or agent knows where to go without parsing all the HTML.
The format is deliberately minimal: a title with the site name, a summary blockquote, then sections of links pointing to clean Markdown versions of the important pages. A variant, llms-full.txt, goes all the way and inlines the entire content into a single file.
The problem it addresses is real: context windows are still finite, and a modern HTML page — nav, cookie banners, JS — is a token sink for an agent. llms.txt is the table of contents you hand it.
What does llms.txt actually do?
One thing: navigation. At inference time — the moment an agent visits your site to accomplish a task — the file hands it a curated map instead of a blind crawl.
Real-world adoption confirms that scope. Niche at launch, it took off when Mintlify rolled llms.txt out across every docs site it hosts in November 2024: overnight, thousands of documentation sites — including Anthropic's and Cursor's — exposed one. The dominant pattern in 2026 is still that one: technical docs consumed by code assistants and agents. Not marketing sites harvesting citations.
What doesn't llms.txt do?
llms.txt blocks no crawlers — it has no power to forbid anything; that's robots.txt's job. It guarantees no citations in ChatGPT, Claude, or Gemini. And it is not an adopted standard: it's a proposal, with no public commitment from any major model provider.
The misunderstanding comes from the name. ".txt at the root" evokes robots.txt, so it sounds like a control mechanism. It's the opposite: robots.txt is imperative (directives that bots follow — or don't), llms.txt is purely declarative. Nothing in the file forbids, allows, or prioritizes anything. A crawler that wants to ignore your llms.txt ignores it; a crawler that wants to crawl your site never needed it in the first place.
llms.txt is a map handed to agents — not a tollbooth, not a no-entry sign, and certainly not a citation ticket.
What does Google say about llms.txt?
Google is the only major player to have taken a public position, and that position is unambiguous. John Mueller confirmed in 2025: "as far as I know, no AI system currently uses llms.txt" — comparing the file to the meta keywords tag, the one search engines have ignored for twenty years precisely because it's self-declared, and therefore gameable.
The anecdote that says it all: in December 2025, an llms.txt briefly appeared on Google's own developer docs — added automatically by their CMS — before being pulled the same day. Even at Google, the file existed without anyone consuming it.
As for OpenAI and Anthropic: no solid public evidence that their systems consume llms.txt for retrieval or citation selection. Your logs may show crawler hits on /llms.txt — a hit is not usage.
Where does llms.txt fit in the robots.txt / pay-per-crawl / RSL stack?
The right way to judge llms.txt is to put it back in its place within the stack of protocols governing the site ↔ AI relationship in 2026. Each layer has a distinct job — and llms.txt holds the most modest one.
Sources: llms.txt proposal (Answer.AI, Sept. 2024); John Mueller statements (Google, 2025); Cloudflare (pay-per-crawl; Markdown for Agents, Feb. 2026); RSL Collective (Sept. 2025).
Two recent moves clarify the trajectory. On one side, monetization is industrializing: Cloudflare's pay-per-crawl answers non-paying crawlers with an HTTP 402, and RSL — launched in September 2025 with Reddit, Yahoo, Medium, and Quora — declares licensing terms directly in robots.txt. On the other, navigation is moving into the infrastructure: in February 2026, Cloudflare launched Markdown for Agents, which converts any HTML page to Markdown on the fly the moment an agent asks for it — up to 80% fewer tokens, per Cloudflare — paired with Content Signals that declare permitted uses (training, search, inference).
The honest read: the de facto standard is being built in the infrastructure, not in a declarative file. And the access layer — open up, lock down, or charge — deserves its own article: blocking AI crawlers is the most expensive GEO mistake of 2026.
Should you publish an llms.txt in 2026?
Yes — as long as you're clear-eyed about what you're buying. The reasoning is a classic asymmetric bet: near-zero cost (an hour of curation, auto-generated on most docs platforms), no downside, and free upside if the standard ever gets consumed. You don't publish llms.txt because you have evidence; you publish it because not doing so saves you almost nothing.
- +Near-zero cost: an hour of curation, auto-generated on most docs platforms
- +No known downside: worst case the file is ignored, best case it gets read
- +Already consumed by some dev tools (code assistants, agents that read docs)
- +You're ready if the standard takes off — the option costs nothing
- −No public evidence of impact on ChatGPT, Claude, or Gemini citations
- −Google has confirmed it doesn't use it (John Mueller, 2025)
- −A stale file misinforms agents — maintenance is mandatory
- −Placebo effect: checking the llms.txt box and believing your GEO is done
How do you publish a clean llms.txt?
- 011 — GenerateOn a docs platform (Mintlify and friends), the file is generated automatically. Otherwise, by hand: a title, a summary blockquote, sections of links. An hour, no more.
- 022 — CurateKey pages only: docs, API, pricing, product pages. An llms.txt is not a sitemap dump — it's an editorial selection for a reader with a finite context window.
- 033 — Serve clean MarkdownEvery link should point to content an agent can actually read: a .md version or minimal HTML. A table of contents pointing to unreadable pages is useless.
- 044 — MaintainAdd the file to your release checklist. A stale llms.txt is worse than none: it misinforms the few agents that do read it.
Is llms.txt enough to get cited by ChatGPT?
No — and this is the most important point in this glossary entry. LLM citations aren't declared; they're earned. Models choose who to cite based on the demand signal: the mentions and searches orbiting a brand across the sources they actually consult — Reddit, comparison sites, press, forums. A self-declared file at the root of your own site has never been part of that equation, for the exact reason Mueller invokes: any signal 100% controlled by the publisher ends up ignored. The full mechanism is unpacked in how LLMs choose who to cite.
llms.txt in a GEO strategy
That's exactly the separation Rankfeed draws: on one side, measure your real citation rate in ChatGPT, Claude, Grok, and DeepSeek; on the other, activate it by generating demand signal across your semantic cluster. The mechanics are detailed on how it works.
llms.txt takes an hour to publish; it won't get your brand cited. If you want to know where you stand — and act on it — Rankfeed measures your visibility across 4 models and activates the demand signal that gets brands cited. See pricing.
FAQ
Keep reading
- Blocking AI crawlers: the most expensive GEO mistake of 2026 — the access/monetization layer of the stack.
- GEO, AEO, LLMO: the dictionary — placing llms.txt within the AI visibility vocabulary.
To measure, then activate, your AI visibility: How it works.