What Is llms.txt? The File, the Format and Whether It Helps

llms.txt is a Markdown file placed at the root of a website, at /llms.txt, that tells AI assistants what the site is about and which pages matter most. The proposal was published in September 2024 by Jeremy Howard of Answer.AI. The idea is simple: large language models have small context windows compared with a whole website, and HTML pages are cluttered with navigation, scripts and ads. A short, curated, plain-text index gives a model the best pages in the format it reads most easily.

It sits alongside two older files that people often confuse it with. robots.txt tells crawlers what they may fetch. sitemap.xml lists every URL for search engines. llms.txt does neither. It is not a permission file and it is not exhaustive. It is a reading list with a summary, written for a machine that is about to answer a question about you.

The proposal keeps the structure small so that both humans and simple parsers can read it. A valid file has these parts in this order.

• An H1 heading with the name of the site or project. This is the only required element.
• A blockquote directly under it with a one or two sentence summary of what the site is and who it is for.
• Optional paragraphs with any context a model needs to interpret the links: what the product does, which audiences it serves, how the docs are organized.
• H2 sections that each hold a Markdown list of links. Each item is a link followed by an optional colon and a short note explaining what the page contains.
• An "Optional" section at the end for secondary links a model can skip when context is short.

A companion file, llms-full.txt, is sometimes published next to it. Where llms.txt is the index, llms-full.txt is the full text of the key pages concatenated into one document, so a model can ingest everything in a single fetch. Documentation sites use it most. For a marketing site, the index alone is usually enough.

Be clear about the limits before you spend an afternoon on it. As of this writing, no major AI company has formally committed to fetching llms.txt as part of how its assistant answers questions. Some crawlers request it, some developer tools and documentation platforms generate and consume it, and it is widely adopted among software projects. But it is not a ranking signal, it does not force a citation, and publishing one will not make ChatGPT or Gemini mention you if your pages are blocked, thin or inconsistent.

What it does well is remove ambiguity cheaply. A model or an agent that does read it gets your preferred description of the business, your canonical product pages and your pricing in one request, without parsing a cookie banner. It is also a useful discipline: writing the file forces you to decide which twenty pages define your site and how you describe yourself in one sentence. That clarity tends to leak into the rest of the site, which is where the visibility actually comes from.

Treat it the way you treat structured data. It is low cost, it helps the readers that use it, and it is a small part of a larger job. Brainito's AI SEO readiness check looks for the file alongside crawler access, extractability, entity clarity and schema, and that order reflects the relative weight of each.

Start with the summary, because it is the part most likely to be quoted. Write it the way you would introduce the company to a journalist: what you are, who it is for, where you operate. Avoid slogans. Then pick the pages. A good marketing llms.txt typically has fifteen to thirty links in four or five sections.

• Products or services. One link per offering, with a note that says what it does and for whom.
• Pricing. The pricing page, with the model in the note, for example "credits, plans from $20 a month, cancel any time".
• Company. About, contact, locations, leadership.
• Help or docs. The pages that answer how-to questions, since these are the questions assistants are asked most.
• Optional. Blog index, case studies, legal pages.

Keep the notes factual and short. "Marketing audit: scores a website across 77 factors and returns a prioritized fix list, free" is a good note. "Our award-winning audit experience" is not. Use absolute URLs. Update the file when you launch or retire a page, and check it at least quarterly.

The file must be served at the root of the domain as plain text, so https://example.com/llms.txt, with a text/plain or text/markdown content type. On most hosts that means dropping it into the public folder. On Next.js, put it in /public, or generate it from a route if your page list changes often. Do not block it in robots.txt, and do not put it behind a redirect to a marketing page, which some sites do by accident when everything unknown redirects to the home page.

To check it, open the URL in a browser and confirm you see raw Markdown, not a rendered page or a 404. Then fetch it with a command line tool and confirm the status code is 200 and the content type is text. If you want to skip the hand-writing, Brainito's llms.txt generator builds the file from a site's pages and lets you edit the summary and notes before you download it.

Does llms.txt replace robots.txt or a sitemap?

No. robots.txt controls access, sitemap.xml lists URLs for search engines, and llms.txt describes and prioritizes pages for AI readers. A site that cares about AI visibility has all three, and they should not contradict each other.

Will it stop AI companies from training on my content?

No. It has no permission semantics. Training opt-outs are expressed in robots.txt with the training crawlers' user agents, such as GPTBot and Google-Extended.

Is it worth it for a ten-page site?

Yes, because it takes twenty minutes. Write the summary carefully, list the ten pages with one-line notes, and move on to the work that moves citations: access, quotable content and a consistent entity.

How is it different from llms-full.txt?

llms.txt is the index. llms-full.txt is the full content of the key pages in one Markdown file. Publish the index first. Add the full file if you have substantial documentation that assistants are asked about.