Understanding llms.txt: A New Standard for AI-Friendly Websites

What is llms.txt?

Introduced by Jeremy Howard in September 2024, /llms.txt is a proposed standard file that websites can include to provide large language models (LLMs) with easily accessible, structured information about their content. Think of it as similar to robots.txt or sitemap.xml, but specifically designed for AI systems to better understand and interact with websites.

Why Do We Need llms.txt?

LLMs face a significant limitation when processing websites: their context windows are too small to handle most websites completely. Additionally, converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is challenging and imprecise.

The llms.txt standard addresses these problems by providing:

• A concise overview of the website's purpose and content
• Links to key markdown documents that LLMs can easily process
• A structured format that both humans and machines can understand

The Format

The llms.txt file uses Markdown format and should be placed at the root of a website (e.g., example.com/llms.txt). It follows this specific structure:

1. An H1 heading with the name of the project or site (required)
2. A blockquote with a short summary of the project
3. Optional paragraphs, lists, etc. with more detailed information
4. Sections with H2 headers containing "file lists" of URLs where further detail is available

Here's a simple example:

# My Company Website > This is the official website for My Company, a tech startup specializing in AI-driven solutions for healthcare. ## Key Documentation - [Our Products](https://example.com/products.md): Detailed information about our AI healthcare solutions - [Company Information](https://example.com/about.md): Our mission, team, and history ## Optional - [Technical Specifications](https://example.com/tech-specs.md): Detailed specifications for developers

The "Optional" section has special meaning—URLs provided there can be skipped if a shorter context is needed, making it perfect for secondary information.

How llms.txt Works with Clean Markdown

The standard also proposes that websites should provide clean markdown versions of their content by appending .md to the URL of the original page. For example, if example.com/products is your HTML page, you would also create example.com/products.md with the same content in markdown format. For pages without filenames, append index.html.md instead.

Benefits for Website Owners

Implementing llms.txt offers several advantages:

• Better AI understanding of your content: LLMs can more accurately interpret and represent your website
• Improved AI search visibility: As AI-powered search grows, this could become an important ranking factor
• Control over AI representation: You decide what information is most important for AI systems to know about your site
• Future-proofing: Position your site for the growing ecosystem of AI tools and assistants

Use Cases

The llms.txt standard is versatile and can serve many purposes:

• Software documentation: Help developers find information about your software
• Business websites: Outline your organizational structure and services
• E-commerce: Explain products, policies, and purchasing processes
• Educational institutions: Provide course information and resources
• Personal websites: Answer common questions about your CV or portfolio

Implementation Tips

If you're ready to create your own llms.txt file, consider these guidelines:

• Use concise, clear language
• Include brief, informative descriptions with your links
• Avoid ambiguous terms or unexplained jargon
• Test your file with different language models to see if they can accurately answer questions about your content
• Consider using a tool like llms_txt2ctx to expand your file into an LLM context file

The Future of llms.txt

The llms.txt specification is still evolving, with community input shaping its development. A GitHub repository hosts the specification, allowing for version control and public discussion. There's also a community Discord channel for sharing implementation experiences and discussing best practices.

As AI continues to transform how we interact with the web, standards like llms.txt will become increasingly important for ensuring that your content is properly understood and represented by intelligent systems.

Getting Started

Ready to make your website more AI-friendly? Here's how to get started:

1. Create a basic llms.txt file following the format outlined above
2. Place it at the root of your website
3. Add markdown versions of your key pages
4. Test with various AI tools to ensure your content is being properly interpreted
5. Submit your website to the llms.txt directory to increase visibility

By implementing the llms.txt standard today, you're taking an important step toward optimizing your online presence for the AI-driven future of the web.