llms.txt
A Markdown overview of a site for LLMs and agents
A proposed file at the root of a site (or under a subpath) that tells LLMs and agents, in Markdown, what the site is and where its most important pages are: /llms.txt.
llms.txt is a proposal (not a formal standard) that Jeremy
Howard published at llmstxt.org in September 2024. The idea is simple:
HTML pages are built for people; turning their content back into clean
text from between menus, ads and JavaScript is hard, and context windows
are limited. So a site publishes a short, hand-curated "table of
contents" written in Markdown at /llms.txt.
Per the spec, the file follows this order: an H1 with the site or
project name (the only required part); a blockquote with a short
summary; free-form paragraphs without headings; and "file lists"
separated by H2 headings. Each list item is a [name](url) link,
optionally followed by : and a short note. By convention an
## Optional heading holds links an agent can skip when context is
tight.
v2, published in August 2026, clarified three things: a file can
live under a subpath (/docs/llms.txt covers only /docs/, and the
most specific file wins); a clean Markdown version of a page can live
at page.html.md or page.md; and both can be made discoverable via
the rel="alternate" type="text/markdown" and rel="describedby" link
relations (as HTML <link> elements or an HTTP Link: header). The
common llms-full.txt (all content concatenated into one file) is not
part of the spec; it is a convention popularised by documentation
platforms such as Mintlify.
Important caveat: llms.txt does not guarantee rankings or citations in AI answers. Google states plainly that AI Overviews and AI Mode need no "new machine readable files, AI text files, or markup". Today its clearest benefit is for coding agents reading your docs and assistants that open pages on a user's behalf.
Picture a researcher visiting a huge library for the first time.
robots.txt is the guard at the door: "you may enter these sections,
not those." sitemap.xml is the full inventory of every book: complete,
but not something you read in an hour. llms.txt is the one-page note
the reference librarian hands you: "This is a library about X; start
with these three books, details are on that shelf, and if you have time
left, these."
The note doesn't let anyone in, and it doesn't make any book more important; it just helps a reader in a hurry find the right shelf faster.
A software library's documentation site publishes /llms.txt: the
library name in the H1, a one-sentence definition in the blockquote, an
"Important notes" paragraph ("the API changed in v3, don't use old
examples"), then link lists under ## Docs, ## Examples and
## Optional. Every link points to the page's .md version.
When a developer tells a coding agent "build a CRUD app with this
library", the agent reads /llms.txt first and opens the two relevant
.md pages, working with a few thousand tokens of clean Markdown
instead of parsing menu-laden HTML.
This site (AI Atlas) itself serves a /llms.txt generated dynamically
from its terms and guides.
# Example Project
> Example Project is an open-source Python library that does X for Y.
Important notes:
- The API changed in v3; v2 examples will not work.
- All links point to Markdown versions of the pages.
## Docs
- [Quick start](https://example.com/docs/quickstart.md): Installation and a first example
- [API reference](https://example.com/docs/api.md): Every function and parameter
## Optional
- [Changelog](https://example.com/changelog.md)# Does the file exist, and is it served as text/plain or text/markdown?
curl -sI https://example.com/llms.txt | grep -i -E "^(HTTP|content-type)"
# v2: does a page advertise its Markdown version and covering llms.txt via Link?
curl -sI https://example.com/docs/page.html | grep -i "^link:"
# Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"
# Which bots actually request it? (nginx access log)
grep "GET /llms" /var/log/nginx/access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn- You publish documentation, an API reference or a developer product; coding agents really do read these files
- Your site has clear pages (product, pricing, policies) that agents look up on a user's behalf
- You can already produce clean Markdown versions of your pages (static site, docs platform, CMS plugin)
- As a cheap, reversible experiment: a few hours of work and low maintenance
- As the fix for Google rankings or AI Overviews visibility; Google says it is not needed
- While classic SEO basics (crawlability, indexing, good content) are missing; do those first
- If you can neither keep it updated by hand nor generate it; a stale llms.txt can be worse than none
Treating llms.txt as a ranking factor
Google's AI features documentation says no extra files or markup are needed; John Mueller said in June 2025 that 'no AI system currently uses llms.txt'. No major answer engine has announced using llms.txt as a citation signal.
Confusing it with robots.txt
llms.txt neither grants nor blocks access. Managing bots is robots.txt's job; llms.txt is only a content map.
Dumping the whole site into one file
The spec asks for a short, curated overview. A list of thousands of links is a sitemap's job; leave the detail to the linked .md pages.
Assuming llms-full.txt is standard
llms-full.txt is not in the llmstxt.org spec; it is a convention from docs platforms. For large documentation it can run to tens of megabytes and fit in no context window.
Deciding without measuring
Check your server logs for which user agents request /llms.txt and the .md pages. If nobody asks, that is data too; the ones that do will mostly be coding agents and tools.