What Is an llms.txt File and Do You Need One?

Every few months a new file lands in the SEO conversation and gets treated as the thing that will finally unlock AI visibility. The llms.txt file is the current one. It arrives with a familiar pitch: put this in your root directory and the assistants will understand your business properly. Nearly every article you will read about it either repeats that pitch or dismisses the file entirely, and both readings miss what is actually going on.

Here is the version we give clients who ask, and we get asked about this most weeks now. The llms.txt file is a genuinely sensible idea with a real problem to solve, published by an independent developer rather than by any company whose systems you care about, and with no confirmed adoption by the ones that matter. It is also cheap enough that the calculation is easy. What follows is what the file is, what belongs in it, what it will not do for you, and how to decide whether yours is worth writing. This sits inside the broader AI search optimization work we run for clients, and it is one of the smaller pieces of it.

What is an llms.txt file?

The reasoning behind it is sound, and it helps to understand the problem before judging the solution. A modern web page is mostly not content. It is navigation, cookie banners, scripts, tracking, styling, related post grids, and footers. A human eye filters all of that instantly. A language model working with a limited context window does not have that luxury. It has to take in the whole document and find the substance inside it, and on a heavy page the substance can be a small fraction of what it receives.

An llms.txt file is a proposed shortcut past that problem. Instead of asking a machine to reconstruct what your company is from twelve rendered pages, you hand it one clean markdown document that says so directly. The convention was proposed by developer Jeremy Howard in September 2024, and the format is deliberately simple: an H1 with your site or company name, a blockquote with a one sentence summary, some optional detail, and then linked lists of your important pages grouped under H2 headings.

The word to hold onto is proposed. This is a convention that some publishers have adopted voluntarily, not a specification anyone maintains or enforces. That distinction gets lost constantly in the coverage, and it is the whole basis for deciding how much effort to put in.

Do AI crawlers actually read llms.txt files?

This is where most articles on the subject quietly overstate. The pattern is easy to spot once you know it. A post asserts that AI systems use llms.txt to understand sites, cites nothing, and moves on to the implementation steps. The claim feels reasonable because the file was built for exactly that purpose, and feeling reasonable is doing all the work.

What we can actually say is narrower. The format has real adoption among documentation sites and developer tools, some of which fetch it deliberately. What is missing is a public statement from any of the major model providers that they consume it when generating answers. We watch for it, and as of this writing there is nothing to point to.

That absence should change how you spend your time, not whether you publish. If llms.txt were a confirmed input, it would deserve a serious content project and ongoing investment. Since it is not, it deserves an hour, done well, and then your attention back on the things with evidence behind them. The signals we can actually observe moving citations in ChatGPT and Perplexity are the substance of your content, the clarity of your entity information, and being mentioned on sources those systems already trust. An llms.txt file is a supplement to that work. It is not a substitute, and we would not let a client treat it as one.

What actually goes in an llms.txt file?

The best way to understand the format is to write one for your own business and notice how hard the first paragraph is. We publish an llms.txt file on this site, and the useful part of building it was not the markdown. It was being forced to state, in one sentence, what the company is, and then discovering that three of our own pages described it slightly differently.

A workable structure looks like this. Open with the company name and a blockquote summary that would survive being quoted back at you by a stranger. Follow with a paragraph of context covering what you do, who you serve, where you operate, and anything that genuinely distinguishes you. Then group your links: key pages, services, case studies, contact. Each link gets a short description explaining why someone would open it.

Three things matter more than the formatting. Be specific, because vague positioning reads as noise to a machine exactly as it does to a buyer. Be accurate about scope, since a company serving clients nationwide should say so plainly rather than leaving a system to infer a local footprint from an address. And keep the same facts you state everywhere else, because contradicting your own service pages gives a machine two versions of you and no basis for choosing. That consistency requirement is the same one that makes schema markup worth implementing properly, and the two work on the same underlying problem from different angles.

How is llms.txt different from robots.txt and your sitemap?

The confusion here is worth clearing up, because it drives a bad decision. People assume llms.txt is a replacement for something, then either skip it as redundant or, worse, treat it as a substitute for technical fundamentals that are actually doing the work.

Robots.txt is a permissions file. It tells crawlers what they may and may not fetch, and it is the correct place to make decisions about AI crawler access, since that is an access question. An llms.txt file grants no permissions and blocks nothing.

Your sitemap is a completeness tool. It exists so a crawler can discover every URL, with no opinion about which of them matter. An llms.txt is the opposite by design. It is a short, opinionated list of the pages you would want read first, and its value comes from what you leave out.

Schema markup describes the page it sits on in a structured vocabulary. An llms.txt describes the site as a whole in prose. Nothing else in your setup does that job, which is the one honest argument for the format that does not depend on adoption. It is the only file where you get to state what your company is, in your own words, in a format built to be read rather than rendered. That framing is also why it belongs alongside the rest of your LLM visibility work rather than in a pile with technical SEO tasks.

Do you need llms-full.txt as well?

We publish both, and the honest report is that the short one did the useful work. The full version took considerably longer to assemble, needs updating whenever the underlying pages change, and its benefit is entirely theoretical until adoption is confirmed. If you are choosing one, choose the short one and write it well.

The case for the full file is real but narrow. If your genuine expertise is distributed across dozens of pages, and any single page tells only a fragment of the story, then a consolidated document lets a system take in the whole picture without crawling all of it. Documentation sites fit that description almost perfectly, which is why they adopted the format first. Most service businesses do not, because their core claims sit on a handful of pages that are easy to reach anyway.

If you do build one, budget for maintenance rather than treating it as finished. A full file that has drifted out of sync with the pages it duplicates is a liability, and the drift is invisible because nothing on your site will ever surface it.

Who should actually bother with this?

The calculation is not complicated once you separate the file from the hype. It costs an hour for a business of normal size. There is no penalty risk, since publishing a factual summary of your own company cannot be construed as manipulation. The upside is unconfirmed but real, and there is a secondary benefit that arrives regardless of whether a single machine ever reads the file.

That secondary benefit is the part worth arguing for. Writing an llms.txt forces you to produce one canonical statement of what your company does, who it serves, and where the authoritative information lives. In our experience doing this for clients, that exercise reliably turns up contradictions nobody had noticed: a service page describing a market the about page contradicts, a positioning line that changed two years ago and survives in four places, a phone number that no longer routes anywhere. Those contradictions were already costing clarity across every system reading the site, and the file is just what made them visible.

The businesses that gain most are the ones hardest to categorize from the outside. A single service company in a single city gets described consistently by the web without help. An agency running production, media, and digital services for clients across the country has genuine ambiguity to resolve, and the same is true for manufacturers with multiple product lines and any firm whose name gives no clue what it sells.

The one group who should skip it is anyone who will not maintain it. An abandoned llms.txt is not neutral. It is a clean, confident, wrong description of your business, formatted for easy consumption, and that is a strictly worse position than staying quiet.

What should you not expect it to do?

The failure we want clients to avoid is not publishing a useless file. It is publishing a useful file and then believing the AI visibility problem is handled. That belief is expensive, because it stops the work that actually matters at the exact moment it was about to start.

An llms.txt file makes you easier to understand. It does nothing to make you worth quoting. If your service pages say what every competitor says, if your content restates what already ranks, if no credible source mentions your company, then a machine that has perfectly understood your site still has no reason to bring you up. Understanding and selection are different problems, and only one of them is solved by a text file.

The practical sequence we run is straightforward. Publish the file, keep it current, and then put your real effort into content with first-hand substance, consistent entity information across every property, and mentions on sources these systems already trust. Then measure your AI visibility properly, so you can see which of those efforts moved anything. If you are still working out how the whole discipline fits together, our breakdown of answer engine optimization and the difference between GEO and traditional SEO covers the ground this file sits inside.

Write the file. Give it an hour, tell the truth in it, put a reminder in your calendar to re-read it each quarter. Then go do the harder work, which is the work that was always going to decide this.