We shipped a feature this week that I've been pushing on for the last six months: automatic llms.txt generation across every customer deployment on the APIMatic developer portal. The technical work was straightforward. The interesting part was how unsettled the surrounding territory still is: what llms.txt actually does, who's reading it, and where the real adoption lives.
Sharing what we learned because most of the public writing on llms.txt is still in the "should I add it to my marketing site" tone. The picture from the inside of a B2B developer-tools company is different.
What llms.txt actually is (and what it isn't)
For the unfamiliar: llms.txt is a markdown file at the root of a domain (example.com/llms.txt) that gives AI systems a structured introduction to the site. The format was proposed by Jeremy Howard in September 2024. Think robots.txt for AI crawlers, but instead of telling bots which paths to avoid, it tells them what this site is and where the high-signal content lives.
The thing it isn't, in mid-2025, is a settled web standard. No major AI lab has publicly committed to reading llms.txt in production. OpenAI hasn't. Anthropic hasn't. Google hasn't. There are credible signals that some are testing it; there's no public commitment.
So why ship it? Two reasons. First, the cost is low. Generating the file is cheap, and the structural discipline of writing one improves the human docs anyway. Second, the audience isn't only the named labs. There's a long tail of AI agents, retrieval pipelines, internal copilots, and crawler-driven products that do read it. The bet isn't "will Anthropic read this next month?". It's "is this file useful to the thousand smaller agents already crawling my customer's docs?"
Who's actually adopting it
I went and looked at who's shipping llms.txt in production by mid-2025. The list is more interesting than the public discourse suggests:
- Anthropic: yes, the company whose models would be the primary reader, ships one.
- Stripe: points the AI crawler at their full API reference.
- Cursor: uses it to surface their docs to other code-gen tools.
- Cloudflare: across their product family.
- Vercel: full developer-platform surface.
- Mintlify: and they ship it as a feature in their docs platform.
The pattern is clear: developer-tools companies first. Companies whose customers integrate through them, not with them. The thinking, at least in our case, was: if developers are increasingly using AI agents to integrate with APIs, the API platform that makes itself most legible to those agents wins.
What surprised us about the implementation
Generating it from existing structured sources is the easy half. Customer SDKs, API references, code samples, getting-started guides; all of it is already in our generation pipeline as structured data. Producing llms.txt and llms-full.txt from that data was a couple of hundred lines of templating.
Picking what NOT to include was the hard half. llms.txt is supposed to be a curated index, not a dump. The whole point is that AI consumers can ingest a small structured file and then follow links to the high-signal subset of the docs. The temptation is to throw everything in. We had to be deliberate: which sections should an AI agent see first, which are noise, which conflict with what the human docs say.
We ended up with three sections in the default template: a one-sentence summary of what the API does, a short list of "featured operations" (most-integrated endpoints), and a longer "Optional" section linking to auth setup, error model, and SDKs. Anything more felt like padding.
The verbose companion file matters more than the index. llms.txt is the index. llms-full.txt is the verbose version: full prose, atomic facts, Q&A blocks. We initially treated llms-full.txt as a "maybe later". In testing, we noticed agents tended to fetch the full file directly and use it as RAG context. Got us to ship both at once.
What we got wrong (early)
For the first month of development, we treated llms.txt like SEO copy: keyword-dense, lots of phrases about "AI agents" and "developer experience". Then we ran a small experiment: gave a few agents both the keyword-dense version and a denser-facts version (fewer adjectives, more atomic claims like "rate limit: 100 requests/minute"), and asked them to answer queries about the customer's API.
The atomic-facts version produced materially more accurate agent responses. The marketing-flavoured version produced confident hallucinations because agents pattern-matched off the adjectives.
So llms.txt rewards dense, declarative, citable claims, and punishes hedged marketing language. Writing for AI consumption isn't writing shorter human copy. It's stripping out every adjective an agent could mistake for a fact.
Where I think this lands in 12 months
My honest read: llms.txt as a hard standard probably doesn't matter by mid-2026. The big AI labs will either build their own retrieval primitives, agree on something more formal, or read every file the same regardless of the name. The exact filename ceases to matter.
What matters, and will outlast the file, is the discipline of writing a structured, AI-readable description of your product. Whether that lives at llms.txt, an MCP server, a JSON-LD blob, or some next-gen spec we haven't named yet, the underlying move is the same: make yourself legible to non-human readers.
That's GEO, in production. It's the thing I was writing about as a thesis last year and the thing we just shipped as a feature. The thesis stays the same. API discovery is going to be mediated by agents, and the teams that don't write for those agents get less of it.
Further reading
- llms.txt: the proposal (Jeremy Howard, September 2024): the canonical spec page.
- llms-txt repository (AnswerDotAI): GitHub home of the spec, examples, and discussion.
- Production examples cited above: Anthropic, Stripe, Cursor, Cloudflare, Vercel, Mintlify.
