In June 2025 we turned on automatic llms.txt generation across every customer deployment of the APIMatic developer portal, and I wrote an essay about it. The last section of that essay was a prediction:
My honest read: llms.txt as a hard standard probably doesn't matter by mid-2026.
It's mid-2026. It doesn't. Four seconds of credit for that, and then straight back, because the prediction came true for reasons I never gave. Worse, five months after writing it came a second post that quietly contradicted the first one, and that post is still on this site.
This is the correction. Skin in the game first, because you should weigh the rest against it: APIMatic ships llms.txt generation as a product feature, and I'm the PM who pushed for it.
What I actually claimed
Two posts, and only one of them's aged badly.
Shipping llms.txt at Scale, June 2025, was appropriately sceptical. It said no major lab had committed to reading the file, that there'd been "credible signals that some are testing it" and no public commitment, and that we shipped anyway on two grounds. The cost was low. And the audience wasn't only the named labs, because there was "a long tail of AI agents, retrieval pipelines, internal copilots, and crawler-driven products that do read it." The bet, in that essay's own words, wasn't "will Anthropic read this next month" but "is this file useful to the thousand smaller agents already crawling my customer's docs?"
That was a reasonable bet with a stated uncertainty. Then in November I wrote GEO is the New SEO, and the uncertainty had disappeared:
When an agent has to answer a query in your domain, llms.txt is the structured context it answers from. Early adopters are seeing tangible results. Products with well-structured llms.txt files report being cited more accurately in AI responses and appearing in more relevant contexts.
Read those two sentences next to each other. The first asserts a mechanism as fact. The second reports results, attributed to nobody, measured by no one named. Five months earlier the same site had said nobody was committed to reading the file. Nothing had changed in between except the kind of post I was writing.
The number that settles it
In June 2026 Ahrefs ran the study I should've run. They took all 137,210 domains in their web analytics panel with traffic in May 2026, checked each root for an llms.txt returning 200, verified the response was real markdown so soft 404s didn't count, and then looked at their bot analytics to see who had asked for it.
28% of those domains publish one. 97% of the published files received zero requests in May 2026. Of the roughly 3% that got any traffic at all, 96% of the requests were bots, and the single largest category was SEO audit tools at 21.7%. Not agents. Tools checking whether the file exists, on behalf of consultants selling you the file.
Ahrefs is careful about their own limits, and it's worth copying. Their panel skews technical and SEO-aware, so they call 28% an upper bound on adoption. And they flag that requests are the generous measure: whether a bot acts on what it fetched is a separate question nobody has answered.
A second study, from EZY Research, watched 83 llms.txt-publishing sites inside a 556-domain server-log panel for twelve weeks to 19 July 2026. It asked a simpler question. When a crawler visits, does it go looking?
| Crawler | robots.txt fetches | llms.txt fetches |
|---|---|---|
| Googlebot | 5,125 | 67 |
| GPTBot | 3,990 | 7 |
| ClaudeBot | 3,120 | 9 |
| PerplexityBot | 775 | 0 |
| Meta-ExternalAgent | 172 | 193 |
Three thousand one hundred and twenty against nine. That isn't a standard finding its feet. That's a file nobody is looking for.
Google said it twice, and the second time I should have been listening
Here's the part that stings. John Mueller was asked about llms.txt on Reddit in April 2025, and his answer was reported widely:
AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag.
April 2025. Seven months before I wrote that llms.txt is "the structured context it answers from". The keywords meta tag comparison is the sharp bit, and it's sharp because it's precise: a file in which you declare what your site is about, to a reader who can simply go and look at your site instead.
He was asked again in June 2026 and hadn't moved. "The file has existed for years, yet none of the AI systems use it, what does it mean?" And Google Search Central's AI-optimization guide, updated on 10 July 2026, now states it as documentation rather than as opinion: creating these files "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."
There's a wrinkle worth reporting because it cuts the other way. Chrome's Lighthouse ships an llms.txt audit under its agentic-browsing checks, updated 5 May 2026, on the grounds that without the file "agents may spend more time crawling the site to understand its high-level structure." So Google Search says it ignores the file and Google Chrome recommends it. Both are Google, both are current, and neither one is lying. They're talking about different readers.
The reader that did show up
Which is the half of my original bet that survived, in a form humbling enough that I nearly missed it.
The Ahrefs study found that agents and agentic infrastructure made up 10.5% of all llms.txt requests, more than any other category of AI bot, and that Claude Code out-fetched every AI retrieval bot, every assistant, and every training crawler in their data. Anthropic's own engineering guidance on writing tools for agents, from September 2025, tells developers that "LLM-friendly documentation can commonly be found in flat llms.txt files on official documentation sites."
So the long tail exists. It just isn't a thousand small agents crawling your docs. It's coding agents, and mostly one of them, fetching a file when a developer points them at a library. That's a real use case and it's the one my customers care about, since the whole point of an API developer portal is that somebody is trying to integrate. But it's a much narrower claim than the one I made, and it has nothing to do with whether ChatGPT recommends your product to your buyers.
The claim I can't defend at all
"Products with well-structured llms.txt files report being cited more accurately in AI responses."
In June 2026 MaxAEO tested that directly. They took 326 llms.txt adopters out of 2,400 tracked domains, matched them to non-adopters on category, organic-visibility band and content volume to get 240 matched pairs, and measured citation rates across eight AI surfaces from February to April 2026. Adopters were cited 11.8% of the time. Non-adopters, 11.6%. A difference of 0.2 percentage points, which the authors put inside their own week-to-week noise band.
They're selling AI-visibility monitoring, so weigh the finding accordingly. Everyone in this story sells something, which brings me to the thing I found most uncomfortable while checking my own work: there's no independent research on llms.txt at all. An arXiv full-text search for the term returns nothing. Every number above, mine included, comes from a company with a product adjacent to the answer.
The spec quietly rewrote itself this month
On 10 August 2026 Jeremy Howard shipped a v2 revision of the llms.txt proposal. It's the most honest document in this whole story and almost nobody has read it.
Three changes matter. It adds a discoverability mechanism, rel="alternate" type="text/markdown" and rel="describedby", because the most common question was how an agent finds the llms.txt covering a given page "without guessing." Which is the spec conceding, in August 2026, that agents couldn't reliably find the file.
It deletes the context-expansion tooling, and with it "the special meaning of the Optional section." Optional sections still exist and they don't do anything mechanically any more. My default template had three sections and one of them was Optional, built on semantics that have now been withdrawn.
And it rewrites the background so that it "describe[s] how agents actually use websites, rather than predicting that they might." The changes page says coding agents now "fetch them as a matter of course", which lines up exactly with the Ahrefs finding and with nothing else in the original pitch.
The spec has reorganised itself around the only reader that turned up, which is the correct response to evidence, and it got there first.
Where the energy actually went
My prediction said the labs would build their own retrieval primitives or agree on something more formal. Something did happen, and it isn't what I described.
WebMCP shipped in Chrome 146 in February 2026, co-developed by Google and Microsoft, with Expedia, Shopify, Etsy and Instacart named among the early experimenters. It doesn't describe what a site is. It exposes what a site can do, as callable tools. Mueller, in the same June 2026 answer where he dismissed llms.txt, volunteered that he likes the WebMCP approach because it has "clear goals and processes." Separately, the IETF has an actual chartered working group, AIPREF, with two standards-track drafts on attaching AI usage preferences to content.
The pattern isn't that llms.txt was a bad idea. It's that llms.txt never had an institutional owner, and the two efforts that got traction are backed by a browser vendor and a standards body. One person maintaining a proposal on GitHub, with 100 open issues and fourteen of the last fifteen commits under his own name, is a remarkable amount of influence for one person to have. It isn't a distribution strategy.
Worth watching, because it's the same shape: Cloudflare's Content Signals Policy went out across 3.8 million domains in their managed robots.txt in September 2025. Enormous distribution, and as of mid-2026 no confirmed enforcement by any major crawler, with Google saying publicly that the directive has no effect on it. Publication without a reader, again, this time at four thousand times the scale.
So what do you do on Monday
Keep publishing the file. The cost argument from 2025 hasn't moved: it's cheap to generate, and the discipline of deciding what an agent should see first genuinely improves your human docs. If your buyers integrate with you through a coding agent, you have a confirmed reader, and that's reason enough on its own.
Then go and check what you say about it. Does your marketing site claim llms.txt gets you cited by ChatGPT? Is it sitting on your roadmap as a growth initiative with a number attached to it? Those are the sentences to fix this week, and one of them is probably somebody's objective, so expect the conversation to be harder than the edit.
Three things are worth your time if you keep the file. Point at it properly, using the link relations v2 added, because until this month an agent landing on an arbitrary page of your site couldn't find it. Drop anything you built on the Optional section, because it doesn't carry meaning any more. And read your own server logs for /llms.txt before you argue with a single number above, because your traffic is the only sample that describes your readers.
What to stop doing is measuring it. If you can't name the agent you expect to fetch the file, you don't have a metric. You have a hope with a dashboard around it.
Both original essays stay up with a note on top rather than having the claim edited out, because a correction that erases what it corrects isn't much of a correction.
The narrower lesson is the one I keep relearning, and it isn't really about a file format. I made a careful claim with the uncertainty attached, and then repeated it five months later in a context that rewarded confidence, and the uncertainty fell off on the way. Nobody removed it. It just didn't survive the second telling.
