A lone orange cactus in a desert of engraved dunes, casting a long shadow in the low sun.

Is llms.txt the internet’s least-read file?

The idea behind llms.txt was a table of contents to help AI understand your website. Yet few files have persuaded so many people to spend time on something no one has promised to read. Here, we explore its history and our own study of 772 websites belonging to listed companies, government agencies, municipalities and regions.

What is an llms.txt file?

An llms.txt file is a plain text file at the root of your website, at er-sajt.se/llms.txt. Its purpose is to give the language models behind services such as ChatGPT, Claude and Perplexity a readable overview of your website: a short description of your organisation, followed by links to the most important pages. Think of it as a table of contents written for a machine. It is optional, and no one sees it unless they look for it.

How the file came about

The story begins with a blog post in September 2024. On 3 September, Jeremy Howard, co-founder of fast.ai and founding CEO of the AI lab Answer.AI, published a proposal with the dry title "/llms.txt – a proposal to provide information to help LLMs use websites". Two years later, Google has taken a position and several large-scale studies have been carried out. The question the proposal left open remains the same: is anyone reading the file?

A problem that looked easy to solve

Howard’s starting point was practical. A language model can only take in a limited amount of text at a time (its context window), and an entire website rarely fits. A web page is also full of navigation, adverts and scripts, and extracting clean text from it is, according to the proposal, difficult and imprecise. His solution was simple: let whoever knows the website best choose what should be read.

Here is what the file looks like. Written in Markdown (a simple format for headings and lists), it starts with the website’s name, continues with two sentences about who you are, and ends with a list of links to the pages that matter most, each with a line describing its contents. It replaces neither robots.txt, which tells bots what they may fetch, nor sitemap.xml, which lists pages for search engines. The proposal itself says the file is intended for use when an AI assistant is helping a user, and that it can complement robots.txt.

An example llms.txt for a made-up company: first the name as a heading, then a short description, and last lists of links to the most important pages, each with a line about what it contains.

Then as now, the proposal calls itself just that: a proposal. No standards body backs it, and no search engine or AI service is obliged to read the file.

The platforms take over

Adoption began where the file made sense on paper: developer documentation. In November 2024, Mintlify, which powers documentation for many technology companies, launched support for the file. In a later post, the company said Anthropic had asked it to develop the feature. That is Mintlify’s own account, and Mintlify has a stake in the file’s adoption, but it shows where the idea first took hold.

Then something happened that explains many of the figures we will come to later. Platforms began creating the file for their clients. Today, llmstxt.org lists Yoast, All in One SEO and Wix among the tools that support it. Website owners did not necessarily have to make a decision at all. Updating a plugin was enough.

The question no one could answer

In spring 2025, the question at the heart of the rest of this story emerged: is anyone reading the file?

In a Reddit discussion in April 2025, Google’s John Mueller replied that no AI service had said it used the file, comparing it to the meta keywords tag. In both cases, website owners themselves say what their websites are about, and search engines have ignored the meta keywords tag for years. In June that year, he wrote on Bluesky that no AI system currently used llms.txt. Reporting from a Google conference in Asia in July, Search Engine Journal quoted Gary Illyes as saying the file was not a Google initiative and not something Google planned to adopt.

What was missing was data. No one could yet say how many websites had the file, let alone how many systems were reading it.

Google answers, then raises another question

In December 2025, according to Search Engine Journal, an llms.txt file appeared on Google’s own search documentation website. Mueller responded with a "hmmn", and the file disappeared within hours, probably an automated publication that no one had requested. It shows how easily the file can end up on websites by accident.

The clearest answer came in spring 2026. Google’s guide to appearing in its AI features includes a section on what you do not need to do. It says there is no need to create new machine-readable files, AI text files or Markdown to appear in Google Search. A note adds that creating and maintaining an llms.txt file is perfectly fine, but that it neither harms nor helps a website’s visibility, because Google Search ignores these files.

Yet the same company raises another question. Chrome’s auditing tool Lighthouse has an experimental category for browsers in which an AI agent acts on the user’s behalf, and it includes an llms.txt check. The documentation calls the file an emerging convention that is currently optional, and the check only flags server errors when fetching it. A missing file is not an error. So Google Search ignores the file, while Chrome’s auditing tool looks for it.

What the logs show

Only in 2026 did the figures start to arrive. The answer depends on how you count, and the studies are not directly comparable.

How many websites have one? In June 2026, Ahrefs found a file on 28 per cent of the 137,000 domains receiving traffic in its analysis. But Ahrefs itself calls this an upper limit, because its clients are technically minded and SEO-aware. Common Crawl, which ran its own crawl in July, received a text file in response from 11.7 per cent of randomly selected websites. In Sweden, where we measured 772 websites ourselves, the figure was 7.8 per cent. More on that shortly.

What kind of files are they? Common Crawl examined more than 584,000 files. Roughly two-thirds, 68 per cent, had been generated from a template, and Wix alone accounted for 41 per cent of all files. Only half fully followed the proposed format.

Who reads them? Ahrefs measured requests for the files during May 2026. Of the websites with a file, 97 per cent received none at all. The 3 per cent that did receive requests accounted for all the traffic, around 22,000 requests, of which 96 per cent came from bots. Of those bots, 77 per cent were not AI tools but SEO auditors, technical crawlers and conventional search crawlers. AI bots accounted for around 20 per cent. No AI bots requested the files on websites where they were absent.

Three figures from the 2026 measurements. 97 percent of sites with an llms.txt got not a single request for the file (Ahrefs). 77 percent of the bots that fetched the file were not AI tools (Ahrefs). 68 percent of the files were made by a template (Common Crawl).

Ahrefs itself points out that "fetched" does not mean "read", and the company also sells the tool it used for the study. We cannot prove that this is the internet’s least-read file, because no one has measured them all. But no study shows the opposite either, and 97 per cent without a single request is an unusually clear signal. There is scattered evidence pointing the other way: an SEO consultant who shared logs showing OpenAI fetching his file every fifteen minutes on a few websites, and an analytics company that, according to Mintlify, saw Microsoft and OpenAI fetching the files, without providing figures or a methodology. Against that stands an operations manager responsible for around 20,000 domains who saw no AI bots at all. As far as we have found, OpenAI, Anthropic, Microsoft and Perplexity have not themselves said that their services read llms.txt. Some publish the file for their own developer documentation, but publishing a file says nothing about whether you read anyone else’s.

The picture in Sweden

To find out what the picture looks like in Sweden, we carried out our own study. We selected 820 organisations: companies listed on Nasdaq Stockholm’s Large Cap and Mid Cap, government agencies, municipalities and regions. Of these, 772 had their own website that we could measure, and on 2 November 2026 we checked them all. Here is what we found about llms.txt.

First, how many have one? The answer is 58 of the 744 websites that responded, or 7.8 per cent. But that figure hides a large difference between the groups.

Group

Has llms.txt

Platform-generated file

Custom file

Nasdaq Stockholm Mid Cap

36 of 125 (28.8%)

25

11

Nasdaq Stockholm Large Cap

14 of 125 (11.2%)

3

11

Central government agencies

4 of 187 (2.1%)

0

4

Swedish municipalities

4 of 288 (1.4%)

0

4

Swedish regions

0 of 20

0

0

Mid Cap companies are more than twice as likely to have a file as Large Cap companies, which looks like a sign that they care more. Now look at the rightmost column. Large Cap and Mid Cap have exactly the same number of custom files: 11 out of 125 each, or roughly 9 per cent. The entire difference comes down to files a platform has created for the website owner: 25 for Mid Cap and 3 for Large Cap. Of the 58 files, 28 are standard files from a platform or plugin, and 30 appear to have been written specifically for the website. The share of platform-generated files is a minimum, because we only count patterns we have identified.

Bar chart of llms.txt files among companies on Nasdaq Stockholm. Mid Cap: 36 of 125 companies have a file, 11 of their own and 25 made by a platform. Large Cap: 14 of 125, 11 of their own and 3 from a platform. The same number of own files; the difference is the platform files.

The contents of the files reinforce this picture. In 19 of them, the text is essentially just a reference to other llms.txt files, and 10 introduce the website using a placeholder name such as "Site" or "Hello world" rather than the organisation’s own name.

Who the file turned out to be for

Follow the story to the end and you arrive back where it began. Ahrefs identifies coding tools and agent-based tools as the most likely audience, which makes sense: the file originated in developer documentation, and that is the kind of reader it suits. A coding agent working with a software library can benefit from a clean list of what is available.

For a typical company website, the picture is different. An AI service answering a question about your organisation fetches your regular pages, so their content and robots.txt determine what it sees. You can check the first by reading the page without scripts, and the second by opening the file. That is where AI "understands" your website, not in a list of links.

If you do have a file, two things are worth knowing. Template-generated files are often thin on content: Common Crawl found that All in One SEO generated files without a summary. And because AI reads text as instructions, Common Crawl has found isolated files containing hidden instructions aimed at language models. Treat the file as code, not prose.

What we recommend

Our recommendation is not to spend time on the file for the sake of search rankings. If your platform has a built-in feature that generates it, enabling it takes little effort. But check that the file introduces your organisation with the right name and a proper description, so it does not become another file with a placeholder name. Spend your time on what makes a difference: which AI bots your robots.txt lets in, and making sure your homepage content is present in the page’s code before any scripts run.

Three things you can do in an afternoon:

  • Check your server logs to see whether anyone is requesting /llms.txt at all. That is the only measurement that tells you anything about your website.
  • Read your robots.txt and check which AI bots you allow in or block. That is a decision that makes a difference.
  • Check that your homepage content is present in the page’s code before any scripts run.

Frequently asked questions

Sources

About Webbindex

Webbindex is Pigment’s study of how Swedish websites work for people and AI. Round 8 covers 772 websites belonging to listed companies, government agencies, municipalities and regions. They were crawled on 2 November 2026 by a bot that openly identifies itself and respects robots.txt, and all raw data is retained so the checks can be rerun. A sample of 30 websites produced 0 discrepancies across 140 verifiable results. Read more about Webbindex and the methodology