llms.txt

llms.txt is a Markdown file placed at the root of a website (or in a subfolder) that summarizes the site in a few lines and lists its main pages, so AI assistants and agents can quickly find clean, structured information.

Timothé Merle

TL;DR

Key Definition

llms.txt is a format Jeremy Howard (Answer.AI) proposed in September 2024, with its specification hosted at llmstxt.org. It holds an H1 title, a summary and lists of links to useful pages, preferably in Markdown form. Google says its search features don't need AI files like this. The assistants, agents and LLM tools that do read it get a curated entry point with the noise stripped out.

Importance

Why Does It Matter?

  • An HTML page mixes content, navigation, scripts and ads. An llms.txt file hands the agent clean Markdown that fits in its context window.
  • The file sends the agent straight to your reference pages (pricing, documentation, offer) instead of leaving it to guess from an exhaustive sitemap.
  • Coding agents and assistants that consult documentation use it to find the right reference page. Fewer vague answers about your product.
  • According to the v2 specification on llmstxt.org, thousands of sites publish one, and Chrome Lighthouse checks for it among its agentic browsing audits.
  • The file is quick to create and changes nothing for your visitors or your Google SEO. The risk is low.
  • Writing the summary makes you spell out what your company does and for whom. All your content benefits.

How Does It Work?

llms.txt works as a reading guide, fetched on demand when an agent or assistant needs information about your site to help a user. Unlike robots.txt, it governs no access, and it doesn't inventory the whole site the way a sitemap does.

  1. 1

    The Format Defined by the Specification

    The file is written in Markdown, in a fixed order. Only the H1 title carrying the site name is required. After it come a summary in a blockquote, optional explanatory paragraphs, then H2 sections with lists of links in the [name](url) format, each followed by a short note if needed. A section called Optional holds secondary links, which an agent can skip when space runs short.

  2. 2

    A Minimal Example

    # Acme Payroll > Payroll software for companies with 200 to 5,000 employees, hosted in the EU. ## Product - [Features](https://acme.com/features.md) - [Pricing](https://acme.com/pricing.md) ## Documentation - [Getting started](https://acme.com/docs/getting-started.md) ## Optional - [Blog](https://acme.com/blog.md)

  3. 3

    Location and Markdown Versions

    The file goes at /llms.txt, or under a subpath such as /docs/llms.txt when it only covers that part of the site. The specification also suggests serving a Markdown version of each useful page at the same URL with .md appended, and advertising it with the rel="alternate" type="text/markdown" and rel="describedby" link relations. Some sites also publish an llms-full.txt that bundles all content into one file. The convention is widespread, but it sits outside the core specification.

  4. 4

    What Google and Others Do With It

    Google Search Central writes that you don't need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode. So Google Search does not use llms.txt to rank or cite your pages. The file is worth it for other readers, the assistants, coding agents and LLM tools that fetch it when a user asks them about your site. OpenAI, Anthropic and Google (for Gemini) each publish an llms.txt for their own developer documentation.

Metrics

How Do You Measure It?

  • Check that https://yoursite.com/llms.txt returns HTTP 200 with a text/plain or text/markdown content type, with no redirect to an HTML page.
  • In your server logs, look for requests to /llms.txt and .md files, and note the user agents calling them (Claude-User, ChatGPT-User, Perplexity-User, developer tools).
  • Hand the file to an AI assistant along with a precise question about your offer, then see whether it lands on the right page.
  • Run a recent Lighthouse audit on your site and read the agentic browsing checks that deal with llms.txt.
  • Judge the file by how accurate the answers are from assistants that read your site. Your Google rankings don't depend on it.

Pitfalls

What Are the Common Mistakes?

  • Selling llms.txt as a Google SEO tactic or a ticket into AI Overviews, when Google says it does not need it.
  • Mistaking it for robots.txt and thinking it allows or blocks crawlers. It controls no access at all.
  • Piling in hundreds of URLs with no curation. It turns into a second sitemap and stops being a concise guide.
  • Pointing to heavy HTML pages, or pages behind an anti-bot firewall, that the agent cannot read cleanly.
  • Stuffing the summary with slogans instead of verifiable facts (audience, offer, price, region). A model gets nothing out of it to describe you.
  • Skipping updates after a redesign and leaving dead links and discontinued offers behind.

What Should You Do First?

Follow these steps to get started.

  • Select 10 to 30 reference pages that describe your offer, pricing, documentation and company.
  • Write a factual two or three sentence summary that says what you do, for whom and where.
  • Sort links into readable H2 sections and keep an Optional section for secondary items.
  • Where you can, serve a Markdown version of the listed pages and link to it from the file.
  • Publish the file at https://yoursite.com/llms.txt and check that it returns HTTP 200.
  • Make sure your firewall lets through legitimate AI agents requesting the file.
  • Build llms.txt updates into the publishing process for your important pages.

Examples

Technical Documentation

Typical scenario. An API vendor publishes /docs/llms.txt with links to Markdown versions of its API reference and guides. A developer asks a coding agent to integrate the API. The agent reads the file, opens the right reference page and suggests a call that matches the current version.

Offer-Focused B2B Site

Illustrative example. A SaaS vendor puts its product page, pricing, customer stories and security page in its llms.txt. An assistant fetches the file to answer "is this tool suited to a 1,000-employee company". It lands directly on the page covering that case, not on an old blog post.

Frequently Asked Questions

  • Does Google use the llms.txt file?

    No, not for Google Search. Google Search Central states that no new machine-readable files or AI text files are required to appear in AI Overviews or AI Mode, which run on the regular Google index. So llms.txt changes nothing for your rankings or citations in Google. It's there for the other assistants, agents and tools that read it.

  • Is an llms.txt file useful if Google does not read it?

    Yes, for the assistants, agents and LLM tools that come looking for information on your site. They get a clean Markdown map of your important pages, easier to work with than HTML cluttered with navigation and scripts. The effect shows most on technical documentation and on precise questions about an offer.

  • What is the difference between llms.txt, robots.txt and sitemap.xml?

    robots.txt tells crawlers what they may crawl. sitemap.xml lists every page search engines should index. llms.txt offers language models an annotated selection of pages, without blocking or allowing anything. The three complement each other and sit side by side at the site root without conflict.

  • How do I create an llms.txt file?

    Write a Markdown file with an H1 title naming the site, a blockquote summary, then H2 sections listing your important pages as links. Publish it at the site root and check that it returns HTTP 200. Hikoo's free llms.txt generator produces a first version from your site, to review before publishing.

  • Should llms.txt go in /.well-known/?

    No. The specification recommends /llms.txt at the root or a subpath such as /docs/llms.txt. It set /.well-known/ aside because that folder only exists at the domain root, whereas an llms.txt file describes the path it sits in. Tools expect to find the file at the root.

Sources

  1. 1llmstxt.org (Jeremy Howard) The /llms.txt file
  2. 2Answer.AI /llms.txt, a proposal to provide information to help LLMs use websites
  3. 3Google Search Central AI features and your website

About the author

Timothé MerleTimothé MerleCo-founder of Hikoo, AEO and GEO expert
View LinkedIn profile

Test your site's AI visibility in 30 seconds.

GEO score, technical health, and an overview of your visibility on ChatGPT and Gemini.