Somebody described llms.txt to you as "robots.txt for AI," and something about that didn't quite sit right. Good instinct. The two files share a postcode and almost nothing else.
Here is the straight version. llms.txt is a single Markdown file you put at the root of your site, at yoursite.com/llms.txt, that hands AI tools a short, curated map of your most important pages. robots.txt is the bouncer deciding who gets through the door. llms.txt is the concierge pointing at the good stuff once they are inside. One controls access. The other gives a guided tour.
This piece covers what llms.txt is, how to build one in an afternoon, and the part most guides skip past on their way to the call-to-action: whether anything is actually reading it yet. That last bit matters, because the honest answer changes how much effort the whole thing deserves. We will get to it.
What llms.txt actually is
llms.txt is a plain Markdown file that lives at the root of your domain, the same place robots.txt and sitemap.xml sit. Inside, it names your site, says in one sentence what you do, then lists your most important pages grouped by topic, each with a short description.
The idea came from Jeremy Howard, who proposed the standard in 2024, and the reasoning holds up. Modern websites are a nightmare to parse. Navigation menus, cookie banners, related-post widgets, three separate newsletter prompts fighting for the same corner of the screen. A language model trying to work out what your business does has to wade through all of it. llms.txt skips the wading. It is a clean summary, written for machines, in a format they read without complaint.
Think of it less as code and more as a contents page. There is no plugin to install, no syntax to learn beyond basic Markdown, and no tooling required. A text editor and ten minutes gets you a first draft you would not be embarrassed to publish.
Why it isn't robots.txt, or your sitemap
Three files sit at your site root, and people muddle them constantly. Here is who does what.
| robots.txt | llms.txt | |
|---|---|---|
| What it does | Tells crawlers what to skip | Tells AI which pages are worth reading |
| Default | Allow everything | No default, purely advisory |
| Format | Plain directives | Structured Markdown |
| Audience | Search and AI crawlers | Language models and AI pipelines |
| Effect | Bots that respect it stay out | Models that respect it prioritise what you listed |
Your sitemap is the third file in the trio, and it is the closest cousin. A sitemap lists every URL you want indexed, in XML, for completeness. llms.txt does the opposite. It is short on purpose, written in Markdown, and it leaves things out on purpose. A sitemap says "here is everything we have." llms.txt says "here is what matters." You ship all three. They are not rivals, and one is not a replacement for the others.
The honest bit: is anything actually reading it?
Now the part the breathless guides tend to bury under a stock photo of a robot. As of mid-2026, no major AI company has confirmed it uses llms.txt in production. Not OpenAI, not Anthropic, not Google.
Google has been blunt about it. When Search Engine Land tracked ten sites for ninety days after adding the file, AI crawl behaviour did not move, eight of the ten sites saw no change at all, and one actually dropped. Google's John Mueller put it plainly: none of the AI services have said they use llms.txt, and you can tell from your server logs that they do not even check for it. He has compared the whole thing to the old keywords meta tag, which in SEO terms is roughly the insult you reserve for a haircut from 2004.
So is it pointless? Not quite, and this is where we step away from the doom takes too. The cost of shipping one is close to zero. The standard is sitting there ready to go, a handful of AI developer tools and documentation platforms already read it, and the exercise of deciding which thirty pages genuinely matter is useful on its own. Treat llms.txt as cheap insurance with a tidy side effect, not as a ranking lever. If adoption takes off next year, you are ready. If it does not, you spent twenty minutes and learned which of your pages you would actually defend in a fire.
And the wider direction is not in doubt, even if this one file is. 25.11% of Google searches now trigger an AI Overview, up 57% in a single quarter, and more than 200 million people now start their research inside an AI chat rather than a search box. The broader work of structuring content so models can quote it is not optional. llms.txt may or may not turn out to be one of the surfaces that sticks.
If an agency wants to bill you a separate line item to "implement llms.txt," push back. It is a twenty-minute job, not a deliverable. AI search optimisation is table stakes in 2026, not a premium add-on, and the small files that support it are no exception.
The format, line by line
A valid llms.txt file has a short, opinionated structure. Four parts, in order:
- An H1 with your site or business name. One line.
- A blockquote summarising what you do in a sentence. A model reads this first, so make it clear, not clever.
- H2 sections that group your content by topic, such as Services, Guides, or Pricing.
- Bulleted links under each H2, pointing to your key pages, with a one-line description after each link.
That is the entire specification. A model can read it top to bottom and understand your site in seconds, instead of crawling forty pages to piece the picture together. Plain text, no JavaScript to run, nothing hidden behind a tab or a carousel.
Keep it to the 10 to 30 pages you would genuinely want an AI to know about. Listing everything turns the file back into a sitemap and defeats the point. Listing nothing is worse than not bothering at all.
What to put in, and what to leave out
Prioritise the pages that earn their place. A page belongs in your llms.txt if it meets at least one of these:
- Pillar pages that cover a core topic properly, not in passing.
- Original research or data. Surveys, benchmarks, case studies. The things only you have.
- Service and product overviews. The canonical description of what you actually sell.
- Pricing. AI gets asked about cost constantly, so make yours easy to find and quote.
- Documentation. Especially if you are a software or technical business.
- Genuine points of view. Your strongest opinion pieces, the ones that prove you know the field.
Leave out the filler:
- Time-sensitive or tactical blog posts that will be stale in a month.
- Thin marketing pages with nothing on them, like the five-sentence "About us."
- Anything a human would never intentionally visit, such as tag archives or pagination.
If you are not sure which of your pages count as pillars, that is usually a content problem wearing an llms.txt costume. Our guide on getting found by AI covers how to build pages worth listing, and our piece on making your site impossible for AI to ignore goes deeper on the readability work underneath all of it.
How to ship one this week
Start to finish, this is an afternoon, not a project.
- Inventory your 20 to 30 most important pages. Pull them from your analytics if you are not sure which ones earn their keep.
- Group them into three to six topical sections.
- Write a one-line description for each page. Plain language. Describe what is on the page, not how amazing it is.
- Format the file in Markdown, following the four-part structure above.
- Deploy it at yoursite.com/llms.txt with no login, no redirect, served as text/markdown or text/plain.
Then visit the URL directly to check it loads. If it downloads a file instead of showing in the browser, your content-type is wrong. Beyond that, there is no submission step worth fussing over. The platforms that consume it will find it. The ones that do not would ignore a submission anyway. Re-read the file once a quarter, add new pillar pages as they ship, cut the stale ones, and tighten the descriptions. It is one of the cheaper recurring chores you can keep on top of.
Frequently asked questions
Does llms.txt help SEO or improve my Google rankings?
No, and anyone promising otherwise is guessing. Google has said it does not use llms.txt as a ranking signal, and there is no evidence it affects your position in search results. llms.txt is about making your content easier for AI tools to read, not about climbing the blue links. Keep it separate from your ranking strategy.
Where should the llms.txt file go?
At the root of your domain, reachable at yoursite.com/llms.txt, exactly like robots.txt. It needs to load with no login and no redirect, served as plain text or Markdown. If visiting the URL downloads a file rather than showing it in the browser, fix the content-type to text/plain or text/markdown.
Do AI tools like ChatGPT actually read llms.txt?
Not reliably, as of mid-2026. No major AI provider has confirmed it parses the file in production, and server-log studies suggest most AI crawlers do not even request it. A handful of AI developer tools and documentation platforms do use it. So it is worth shipping as low-cost preparation, but do not expect it to drive traffic on its own yet.
What is the difference between llms.txt and llms-full.txt?
llms.txt is the short, curated index: your key pages with brief descriptions. llms-full.txt is the long version, containing the full text of those pages in one file so a model can read everything without following links. Most small businesses only need llms.txt. llms-full.txt suits documentation-heavy sites where the full content really matters.
Is llms.txt an official standard?
It is a proposed open standard, published at llmstxt.org by Jeremy Howard in 2024, not something ratified by Google or the major AI companies. It has a clear specification and growing informal adoption, but no official endorsement from the big search or AI platforms. Treat it as a sensible convention, not a requirement.
Should I bother with llms.txt if my site is small?
Yes, because small sites get the most out of the exercise. Forcing yourself to name your ten best pages and describe each in a line is a useful clarity check regardless of who reads the result. It costs twenty minutes, it cannot hurt your SEO, and it leaves you ready if adoption picks up. Just keep your expectations about traffic realistic.
Build your llms.txt, keep it tidy, and get on with your day. It is a small, cheap, sensible thing to have, and it is not the thing that gets you cited by AI. That comes from content worth quoting and a site models can actually read, which is the work our AI SEO service does from month one, with pricing on the website and no contracts to escape later. If you would rather see where you stand first, a free SEO and AI audit will show you what AI sees when it looks at your site. And if a previous agency tried to bill you a fortune for a twenty-minute text file, get in touch. We will tell you, free of charge, exactly how much that should have cost.
