Why Your Website Needs llms.txt

A website has a new reader. Not the person who opens a page, notices the large heading, looks at a case study, and understands what its author does within a few seconds. This reader arrives with a question: “Who is this?”, “Which tool should I choose?”, or “Where is the documentation?” Then it tries to assemble an answer from HTML, links, snippets, and whatever files it can access.

That reader is an AI agent. It already searches for information, compares products, reads documentation, prepares drafts, and hands people links. It does not read a site the way we do with our eyes.

A good page is not always legible to an agent

People infer context from form very quickly. We see that “Projects” is a section, a card is a case study, and a large number next to a logo is probably an outcome. We notice tone, recognise when an old post is no longer current, and open another tab when we need more detail.

An agent sees a different version of the same page: text in the DOM, metadata, links, and the content it managed to retrieve in one or several requests. If the answer to an important question is spread across the home page, three case studies, and a social profile, a model may not find it at all—or confidently stitch together an inaccurate summary.

The problem is not that an agent “reads badly.” Visual hierarchy is simply a dense way to communicate meaning to a person. A machine needs more explicit anchors: who is speaking, what they do, which claims can be repeated, and where to go for detail.

What llms.txt is

llms.txt is a short text file at a website’s root. Its idea resembles robots.txt or sitemap.xml: offer a clear entry point for machines. It can name the site, explain its purpose, and link to a handful of useful pages or a fuller profile.

It is important not to oversell the format. It is not a universally required standard, a button that gets you into ChatGPT answers, or a replacement for good HTML, SEO, and documentation. Different agents and crawlers may not look for it or follow every link. But when a tool can read it, the file saves it from having to guess.

On this site, I use two levels:

• llms.txt is the short entry point: who I am, what I work on, and where to find projects, writing, GitHub, and contact details.
• llm.md is the fuller profile: experience, working principles, selected tools, links, and attribution guidance.

This does not compete with the home page. The home page helps a person get the context. The profile helps an agent keep it intact.

What to put in it

A useful file does not try to retell the whole website. It answers the basic questions before the agent starts inventing its own version:

• who the author, team, or organisation is;
• what they do now, and in which context that matters;
• which projects or pages are the sources of truth;
• which wording is appropriate for a short description;
• how to get in touch and which public links are current.

For a portfolio, that can be a professional summary, experience, a few case studies, and contact details. For a product, it can be purpose, key flows, documentation, API references, and a status page. For an open-source project, it can point to the README, installation instructions, supported versions, repository, and licence.

The main rule is to write only what can be verified on the website or in public sources. Skip “market-leading,” “revolutionary,” and other phrases an agent may later repeat as facts. A concrete role, a link, and a clear boundary are better.

This is not only for personal websites

A personal site is a convenient small example, but the idea is broader. Any system an agent needs to work with benefits from explicit context. We already do this in design systems: we do not expect an engineer to infer the meaning of a component from a picture. We specify states, tokens, APIs, and constraints.

A website is a system too. It has entities, current versions, public promises, and routes to sources of truth. If all of that exists only in the author’s head and a polished composition, a person can usually work it out. An agent has to improvise.

So the useful question is not “Does every website need llms.txt?” A small landing page with one form will probably not benefit from another file. Ask instead: Is there important context that a machine should repeat without guessing? If the answer is yes, it is worth assembling.

The file is only the beginning

llms.txt cannot fix a site by itself. It does not replace clear headings, good metadata, working links, accessible HTML, or current documentation. If a website contradicts its own profile, the agent gets more—not less—confused.

But as a discipline, the small file is useful already. It forces you to articulate what a new type of reader should understand about you, a product, or a project. Once that is clear for a machine, it is usually clearer for people too.

We used to design a page so a person could quickly see where to look. Now it is worth considering what an agent should understand when nobody is there to look.