
Key points
llms.txtis a Markdown file at your site root: a title, a one-line summary and lists of your most useful pages with short notes.- It is a community proposal from 2024. It does not block or allow anything, and Google has said Search does not use it.
- It is cheap to add and harmless. Do not expect rankings from it; keep it short, accurate and current.
What it is
Large language models read web pages the way people do, but pages are full of menus, scripts and adverts. The llms.txt proposal, published in 2024 by Jeremy Howard of Answer.AI, suggests a short Markdown file at /llms.txt that lists the pages most worth reading, so an AI tool does not have to guess.
What it looks like
# Example Store
> Example Store sells handmade ceramics and ships across the EU.
## Products
- [Mugs](https://example.com/mugs): all mug designs, sizes and glazes
- [Plates](https://example.com/plates): dinner and side plates
## Help
- [Shipping and returns](https://example.com/shipping): costs, times and the 30-day return policy
## Optional
- [Our story](https://example.com/about): who makes the pieces
- One H1 with the site or project name (the only required part).
- A blockquote summary right under it.
- H2 sections containing link lists:
- [Name](URL): note. - A section called Optional marks links that can be skipped when space is tight.
- Use absolute https:// addresses, because the file is often read away from your site.
llms.txt, robots.txt and sitemap.xml
- robots.txt tells crawlers what they may fetch. It is the place to allow or block AI crawlers such as GPTBot or ClaudeBot.
- sitemap.xml lists every indexable URL for search engines, with no descriptions.
- llms.txt is a short, hand-picked reading list with notes, for AI tools. It controls nothing.
Does anyone use it?
Support is patchy. Some documentation platforms generate it automatically and some developer tools and agents read it. Google representatives have said that Google Search does not use it, and no major AI provider has promised to. There is no reliable evidence that it changes how often you are cited in AI answers.
Spend your effort first on things that definitely matter: fast, crawlable pages, clear headings, structured data and a correct robots.txt. Add llms.txt afterwards if you want to.
How to make a good one
- Write the one-sentence summary as if explaining the site to a stranger.
- List 10 to 30 pages that answer real questions: documentation, pricing, policies, key guides. Skip tag pages, search results and login pages.
- Add a short note to each link that says what the page answers.
- Upload it to the site root so it loads at
https://yoursite.com/llms.txtas text, not an HTML page. - Check it with the llms.txt checker, which also drafts a starting file from your home page. Re-check whenever pages move.
A longer /llms-full.txt with the full text of key pages is an optional companion. Do not put private or paid content in either file: both are public.
See also: HTTP headers explained.
FAQ
Is llms.txt an official standard?
No. It is an open proposal that anyone can implement. There is no standards body behind it and no guarantee any given AI tool reads it.
Will llms.txt stop AI from using my content?
No. It does not block anything. To ask AI crawlers not to fetch your pages, use robots.txt rules for the crawlers you want to exclude.
Can I generate it automatically?
A draft can be generated from your navigation, and many CMS plugins and documentation tools do this. Review the result by hand, since automated lists tend to include pages that do not belong.
How long should it be?
Short. A curated list of 10 to 30 links with notes beats a dump of every URL. Put long content in a separate /llms-full.txt.
Is there any risk in adding it?
Only a small one: it is public and lists your pages, so keep private areas out, and keep it up to date so it does not point at broken links.