llms.txt: What It Is, What It Does, and Whether You Need One

llms.txt is a plain text file, written in Markdown, that lives at the root of your domain, like example.com/llms.txt. Its purpose is to give large language models a concise, human-curated overview of your site: who you are, and links to the pages that matter most, each with a short description.

The idea was proposed publicly in 2024 by Jeremy Howard of Answer.AI. The reasoning is sensible: web pages are full of navigation, scripts and layout noise, and context windows are limited. A clean summary file could help a model find the right content faster.

The basic structure

• An H1 with the site or project name.
• A short blockquote summary of what the site is.
• Optional paragraphs with key context.
• H2 sections with lists of links, each link followed by a one-line description.
• An optional section for secondary resources that can be skipped when context is tight.

People confuse these constantly, so let me separate them cleanly.

• robots.txt tells crawlers what they may or may not fetch. It's an access instruction. If you want to allow or block AI crawlers, this is the file that matters, along with any platform-specific controls.
• sitemap.xml lists URLs you want search engines to discover, usually generated automatically and often containing every indexable page.
• llms.txt offers a curated, readable summary for language models. It doesn't grant or deny access, and it isn't meant to list everything.

This is the chapter most llms.txt guides skip, because the honest answer is less exciting than the hype.

Many developer-focused companies publish llms.txt files, and some AI coding tools and agents can use them when pointed at documentation. That's real, practical usage.

The big AI search and assistant platforms are a different story. Adoption has been unclear and uneven, and Google has publicly said it doesn't use llms.txt for Search, with one of its search advocates comparing it to the old keywords meta tag. Platforms can change their approach, but you shouldn't plan around a benefit nobody has confirmed.

If a tactic takes an hour and might help, do it. If someone sells it to you as the key to AI visibility, walk away.Shmul

Worth doing

• Developer tools and APIs. Documentation sites benefit most, because AI coding assistants and agents are often pointed at docs.
• Large knowledge bases. When you have hundreds of help articles, a curated map of the canonical ones reduces confusion.
• Sites with complex product lines. A clear summary of what each product is can reduce mix-ups.
• Anyone who already has solid fundamentals and wants to cover small bases.

Fine to skip for now

• Small local businesses with a handful of pages.
• Sites that still have crawlability, indexing or content quality problems. Fix those first.
• Teams with no capacity to keep the file accurate as the site changes.

1. 1Write the one-paragraph summary. Who you are, what you offer, who it's for. Plain language, no slogans.
2. 2Pick your canonical pages. Core product or service pages, the most important guides, pricing, about, and key documentation. Usually 15 to 60 links, not 600.
3. 3Group them. Use H2 sections like Products, Guides, Documentation, Company.
4. 4Describe each link in one line. Say what the page answers, not marketing copy.
5. 5Add a skippable section for secondary resources.
6. 6Publish at the root as /llms.txt, served as plain text.
7. 7Put a review date in your calendar. An outdated map is worse than none.

• Dumping the whole sitemap. Thousands of links make the file meaningless. Curate.
• Marketing language. "Industry-leading, best-in-class platform" tells a model nothing. Describe what the page actually contains.
• Linking to redirected or broken URLs. Point to final canonical URLs.
• Setting and forgetting. Products change. The file should too.
• Treating it as access control. It isn't. Use robots.txt for that.
• Expecting it to fix visibility. If you aren't cited today, the reason is almost never a missing text file.

Since llms.txt doesn't control access, it's worth being clear about what does. The practical control for crawling is still robots.txt, together with any platform-specific settings the AI companies document.

• Named user agents. AI companies publish the user agent names their crawlers use, and some separate crawlers for training from crawlers used to fetch pages for live answers.
• Robots.txt rules per agent. You can allow or disallow specific agents, rather than all bots at once.
• Google-specific controls. Google documents a separate token for controlling use of content in some of its AI products, distinct from Googlebot crawling for Search.
• Server-level blocking. Some sites block by IP or at the firewall, which is more forceful but needs maintenance.

Decide what you actually want

Blocking all AI crawlers may protect content from training, but it can also reduce your chances of being cited in AI answers that fetch live pages. Allowing everything maximizes visibility but gives up control. Many businesses allow retrieval for answers and make a separate decision about training.

Here's how I'd rank the work if your goal is showing up in AI answers.

1. 1Crawlable, indexable, fast pages. If systems can't access your content, nothing else matters.
2. 2Clear, specific, answer-first content on the topics you want to be cited for. See getting cited in ChatGPT.
3. 3Consistent third-party mentions on the sources models lean on: reviews, comparisons, publications.
4. 4Entity clarity across your site and profiles. See entity SEO.
5. 5Measurement of citations over time. See measuring LLM citations.
6. 6Nice-to-haves, including llms.txt.

llms.txt is the bow on the present. Make sure there's a present first.Shmul