Skip to content
Get a free audit

AI Search Optimization

llms.txt Explained: What It Is and Do You Need One

Published 8 min read

Add saadrazaseo.com as a preferred source on Google
Cover graphic for the AI search guide "llms.txt Explained: What It Is and Do You Need One" by Saad Raza SEO

llms.txt is a proposed Markdown file at the root of your site that gives AI tools a short, curated map of your most useful pages. Google says you do not need one for its AI features, and I have found no AI search provider documenting that it reads the file to choose citations. Treat it as optional.

Adoption is real but small: the 2025 Web Almanac SEO chapter found valid llms.txt files on 2.13 per cent of desktop sites and 2.10 per cent of mobile sites, and 39.6 per cent of those files were tied to the All in One SEO plugin, which suggests many are generated automatically rather than chosen.

If you run a business site and keep hearing that llms.txt is the new robots.txt, this guide is the sober version. I will explain what the file is, what the specification actually asks for, what Google and Chrome have said about it, and how to decide whether the hour it takes is worth spending on your site. It sits inside the wider topic of generative engine optimisation, and the short answer is that it is a low-cost experiment, not a ranking lever.

What llms.txt actually is

The idea was proposed by Jeremy Howard in September 2024 on llmstxt.org. The reasoning is practical. A language model reading your site has a limited context window, and a normal web page is full of navigation, scripts and layout that waste it. So the proposal asks site owners to publish one plain Markdown file, at /llms.txt, that summarises the site and points to the pages worth reading, ideally with clean Markdown versions of those pages.

Two points are easy to miss. First, it is a proposal and a convention, not an internet standard like robots.txt, and no body enforces it. Second, it is not an access-control file. It does not allow or block anything. If you want to control which crawlers can fetch your pages, that is the job of robots.txt, which I cover in the guide to AI crawlers and robots.txt.

The format, line by line

According to the specification, the file is ordinary Markdown with a fixed order. Only the first element is mandatory.

  1. An H1 with the name of the site or project. This is the only required part.
  2. A blockquote with a short summary. One or two sentences on what the site is and who it is for.
  3. Optional plain paragraphs or lists. Any extra context a model needs to interpret the links, such as how your services are structured.
  4. Zero or more H2 sections containing file lists. Each list item is a Markdown link, optionally followed by a colon and a short note about what the page contains.
  5. A section called Optional. By convention this holds secondary links an agent can skip when it needs a shorter context.

The specification also suggests offering Markdown versions of pages by adding .md to the page URL. That part is where most of the real work sits, and it is the part many plugin-generated files skip.

A simple example file

Here is an illustrative example for an imaginary dental clinic in Karachi. It is not a real site, and the URLs are placeholders.

# Bright Smile Dental Clinic

> Family dental clinic in Karachi offering check-ups, braces, implants and emergency care. Prices are listed in PKR on each treatment page.

## Treatments

- [Dental implants](https://example.com/implants/): process, recovery time and suitability

- [Braces and aligners](https://example.com/braces/): options, typical duration, who it suits

## Practical information

- [Opening hours and location](https://example.com/contact/): address, hours, parking

- [Pricing](https://example.com/pricing/): consultation and treatment price ranges

## Optional

- [Blog](https://example.com/blog/): general dental advice articles

Notice what makes it useful: the notes say what each page contains, and every link points to a page that is already clear and complete. If those pages are thin, no summary file will rescue them.

What is known about adoption and support

This is where honesty matters, because the topic is full of confident claims. Here is what I could verify on official pages.

  • Google Search. Google’s AI optimisation guide lists llms.txt files and other special markup among the things you do not need for generative AI features. Its page on AI features and your website says you do not need to create new machine readable files, AI text files or markup to appear in them. This applies to AI Overviews and AI Mode.
  • Chrome Lighthouse. Lighthouse includes an llms.txt audit in an agentic browsing category and describes the file as an emerging convention. If the file does not exist, the audit is simply marked not applicable. It only flags a server error when fetching the file.
  • The proposal itself. The llmstxt.org page now describes a v2 and says thousands of sites publish the file, documentation platforms generate it, and several AI labs publish one for their own developer documentation. That shows the file exists in the wild. It does not show that AI search products read it when deciding what to cite.
  • OpenAI, Anthropic and Perplexity. Their crawler documentation covers user agents and robots.txt. I could not find any of them stating that llms.txt influences retrieval or citation. Absence of a statement is not proof of non-use, but it means you should not plan around it.

So the evidence is mixed in a specific way. Some tooling and some agents may read the file. The company with the largest search share says you can skip it. Nobody has published evidence that having one earns you more citations.

Cost and benefit, plainly

Factor Reality for most business sites
Effort to publish a basic file Under an hour if your key pages are already clear.
Effort to do it well Higher, because Markdown versions of pages and ongoing maintenance are the real cost.
Risk to SEO Low. A plain text file at the root does not change how Google ranks your pages.
Risk of being wrong Moderate. A stale file that lists deleted or renamed URLs sends agents to dead ends.
Proven citation benefit None that I can point to from an official source.
Possible benefit Helps developer tools and coding agents that read documentation, and may help future agents.

My decision rules

When someone asks me whether to add llms.txt, I do not answer yes or no. I ask three things.

  • Do you publish documentation, an API, a knowledge base or a product with many technical pages? If yes, a file is more likely to be read by developer tools, and the maintenance can be automated from your docs platform. Create one.
  • Are your core pages already crawlable, server rendered and clearly written? If not, fix that first. A map to unclear pages does not help anyone.
  • Do you have someone who will keep it current? If the answer is no, skip it or generate it from your sitemap or CMS so it updates itself.

For a local service business or a small shop with a dozen pages, I would call it optional. Your time is better spent on the foundations in the GEO overview, such as clear answers, consistent entity information and third party mentions. If you want a technical review that covers crawl access and rendering before you worry about files like this, that is part of my technical SEO services.

How to create one without making a mess

  1. List your ten to thirty most useful pages. Choose pages that answer real questions: services, pricing, policies, location and your best guides.
  2. Write one useful note per link. Say what the page contains in plain words, not a keyword string.
  3. Group links under clear H2 headings. Services, guides, policies and company information work for most sites.
  4. Put low priority material under Optional. Blog archives and old news belong there.
  5. Upload it to the root so it loads at /llms.txt. Check that it returns a 200 status and plain text, not an HTML error page or a redirect to your homepage.
  6. Add it to your maintenance routine. Review it whenever you launch, merge or remove pages.
  7. Check your server logs after a few weeks. If nothing ever requests the file, you have learned something useful about how much it matters for your site.

On that last point, measuring beats guessing. If you also want to see whether AI tools send visitors at all, my guide to measuring AI referral traffic in GA4 shows how to separate those sessions, and tracking brand visibility in AI assistants covers how to test what the assistants say about you.

Common mistakes to avoid

  • Treating it as a ranking tactic. Nothing official supports that, and Google says it is not needed for its AI features.
  • Stuffing it with keywords. The file is meant for a reader, human or machine, that wants an accurate map.
  • Listing blocked or noindexed pages. If robots.txt blocks a page, pointing to it creates a contradiction. My explanation of how robots.txt works covers the basics.
  • Claiming Markdown pages exist when they do not. Only link what loads.
  • Letting a plugin publish it unreviewed. Open the file and read it. Plugin output often includes pages you would not choose.

Questions people ask about llms.txt

Does llms.txt help me appear in ChatGPT, Gemini or Perplexity answers?

I have found no official documentation from OpenAI, Google or Perplexity saying so, and Google states you do not need special AI files for its features. Visibility in those tools depends far more on being crawlable, clear and mentioned elsewhere on the web. See my notes on getting cited in ChatGPT search.

Is llms.txt the same as robots.txt?

No. Robots.txt tells crawlers which URLs they may fetch and is widely honoured by reputable bots. The llms.txt file allows or blocks nothing. It is a suggested reading list written in Markdown.

Can llms.txt hurt my SEO?

A correct file is very unlikely to affect your Google rankings. The practical risks are an out of date file that lists broken URLs, or exposing pages you would rather not highlight. Review it like any public page.

Should I create Markdown versions of every page?

Only if you can keep them accurate. The specification suggests it, but duplicated content that drifts out of date causes more harm than good. For most business sites, strong HTML pages with clear headings and server rendered content are a better investment.

If you would like me to check whether AI crawlers can reach your key pages, and whether a file like this is worth your time, you can request a free audit through my contact page.

More AI search guides: start with the generative engine optimization guide, then go deeper: