If you have been following the AI search conversation lately, you have probably seen the term llms.txt come up. Someone in a Slack group posts it. A consultant adds it to a checklist. A thread on X calls it the next big thing for SEO. And then someone else replies that it is completely pointless and no one uses it. Both reactions are a little right and a little wrong.
Here is the straightforward version: llms.txt is a proposed standard for a plain-text file you place at the root of your website (at yoursite.com/llms.txt) that summarizes what your site is and links to the pages that matter most. The goal is to give large language models a clean, structured, machine-readable entry point to your site, rather than forcing them to crawl and interpret dozens of HTML pages on their own.
Think of it as leaving a well-organized briefing packet on your front desk, instead of making a visitor wander through every room to figure out what you do.
Where the idea comes from
The spec was proposed by Jeremy Howard (of fast.ai and Answer.AI) at llmstxt.org. The core insight is simple: HTML is built for browsers. It is full of navigation menus, cookie banners, JavaScript, ads, and boilerplate that adds noise for a model trying to understand what a page is actually about. A clean markdown-formatted plain-text file strips all of that away.
The proposal is not an official standard from any standards body. It is a community proposal that some site owners have started adopting. Browser-vendor support, crawler support, and AI engine support are all patchy and evolving. That context matters when you are deciding how much energy to spend on it.
What the file actually looks like
The spec is refreshingly simple. The file uses markdown conventions. It starts with an H1 containing your site or brand name, followed by a blockquote with a one-sentence summary of what you do. After that, you add H2 sections with markdown-formatted links to your most important pages, with a short description for each link.
llms.txt (example for a fictional analytics product)
# Acme Analytics > Privacy-first web analytics for small SaaS teams. Self-serve, no cookies required. ## Product - [Features](https://acme.com/features): what Acme tracks and how the dashboard works - [Pricing](https://acme.com/pricing): plans, limits, and the free tier - [Changelog](https://acme.com/changelog): recent releases and what changed ## Docs - [Quick Start](https://acme.com/docs/quick-start): install the snippet and verify your first event - [API Reference](https://acme.com/docs/api): REST API endpoints and authentication - [Privacy FAQ](https://acme.com/docs/privacy): data retention, GDPR, and cookie-free tracking ## Company - [About](https://acme.com/about): the team and why we built this - [Blog](https://acme.com/blog): product updates and analytics thinking ## Contact - Website: https://acme.com - Support: support@acme.com
That is it. There is no schema to validate, no JSON to format, no plugin to install. If you can edit a text file and upload it to your server root, you can ship an llms.txt in under fifteen minutes.
How it differs from robots.txt and sitemap.xml
People often ask how llms.txt relates to the existing files you probably already have. Here is a quick comparison.
| File | Primary purpose | Who reads it | What it controls |
|---|---|---|---|
| robots.txt | Access control for crawlers | Search engine bots, any web crawler | Which pages a crawler is allowed or disallowed from fetching |
| sitemap.xml | Page discovery | Search engine crawlers | A structured list of all URLs on your site and their metadata |
| llms.txt | Meaning and context | AI models and LLM-powered tools (in theory) | What your site is about and which pages carry the most important information |
They serve different needs and are not substitutes for each other. robots.txt is about permissions. sitemap.xml is about completeness. llms.txt is about interpretation. You can have all three without any conflict.
The honest debate: does it actually matter?
Here is where I want to be straight with you, because a lot of content on this topic skips past the real uncertainty.
The case for llms.txt
- ✓It is low effort. Writing one takes less time than a single blog post.
- ✓It costs nothing to publish. No tools required, no ongoing maintenance unless your site structure changes.
- ✓It signals intent. Even if AI crawlers do not weight it heavily today, you are establishing a clean, summarized entry point that may become more useful as the spec matures.
- ✓Some AI-powered tools that let users connect external data sources (like custom GPTs, Claude Projects, and Perplexity's focus feature) already look for or benefit from a clean text-format summary of a site.
- ✓The broader principle it embodies is well-supported: structured, machine-legible content is easier for AI systems to interpret and cite. The GEO research paper (Aggarwal et al.) found that optimizing content to be citation-ready and structured can lift a source's visibility in AI-generated answers by up to roughly 40%. llms.txt is one small piece of that larger picture.
The case for skepticism
- ✓No major AI engine (ChatGPT, Gemini, Perplexity, Claude) has publicly confirmed that llms.txt meaningfully influences their answers or crawling behavior.
- ✓The spec is not endorsed by any standards body. Adoption is voluntary and inconsistent.
- ✓LLMs are trained on large web crawls. A single text file at your root is a tiny signal compared to thousands of pages of indexed content.
- ✓If your pages are already well-structured with clean headings, good metadata, and authoritative content, you may already be doing most of what llms.txt would add.
The honest answer is we do not know exactly how much it moves the needle yet. But it is ten minutes of work with no downside. Ship it and focus your bigger energy elsewhere.
Who should bother writing one?
Short answer: most sites probably should, given how low the cost is. But here is a more useful breakdown.
- SaaS products and tools. If someone asks an AI assistant for the best privacy-friendly analytics tool and you want to be in the answer, having a clean machine-readable summary of what you do is a reasonable hedge.
- Consultants, agencies, and solo operators with a clear service offering. Your site might be ten pages, which makes the file trivial to write and arguably more useful since you can cover your whole site in one tight summary.
- Publishers and content-heavy sites. Use the file to highlight your most important categories and cornerstone pieces rather than a full index.
- Sites that are already investing in AEO or GEO. If you are thinking about how AI systems find and cite your content, llms.txt is a natural part of that checklist.
Who can probably skip it for now: sites whose primary audience is not searching for their type of content via AI chat interfaces, or teams that have ten more important things to fix first (page speed, mobile experience, thin content). Get the fundamentals right before worrying about emerging signals.
Tips for writing a good one
- ✓Write the blockquote summary as if you were answering the question what is this site, in one sentence, for a smart stranger. Clear and specific beats clever.
- ✓Link to pages that actually explain what you do, not just your homepage. Docs pages, feature pages, and your pricing page are often more useful than your blog index.
- ✓Keep descriptions on each link short and specific. Plans and pricing tells a model more than learn more.
- ✓Aim for maybe ten to twenty links total. A 200-link dump of every URL on your site defeats the purpose of the summary.
- ✓Update it when your site structure changes, like when you add a major product or restructure your docs.
The bigger picture: being machine-legible
llms.txt matters not as a magic trick but as part of a broader principle: the sites that get cited and surfaced by AI systems tend to be the ones that are structured, specific, and easy to interpret without ambiguity. Clear headings, good metadata, structured data markup, authoritative content, and yes, a well-written llms.txt all push in the same direction.
Want a working llms.txt in under two minutes? The free generator tool below analyzes your site and drafts the file for you, so you can review and ship it rather than write it from scratch.
~40%
Potential visibility lift from structured, citation-ready content in AI answers (Aggarwal et al., GEO paper)
The bottom line
llms.txt is a simple, low-cost signal that tells AI models what your site is about and where to look. The spec is real, the logic is sound, and the file takes no time to write. The skeptics are also right that nobody should expect miracles from a text file: it is one small part of a larger picture that includes good content, structured markup, and genuine authority.
Write it. Keep it honest and specific. Then spend your real energy on the content quality and structure that actually drives whether AI systems find your site worth citing.