A few months ago, I watched a Claude-powered coding agent try to integrate a perfectly well-documented API. The docs were clean. The OpenAPI spec validated. The endpoints did exactly what the names suggested. And the agent still produced code that didn't compile, hallucinated a pagination parameter that didn't exist, and confidently used a deprecated auth flow.
That spec wasn't bad. It was just written for humans.
That gap, between "good API docs" and "API an AI coding agent can use without supervision," is what I've spent the last year working on as the PM for Context Plugins at APIMatic, our MCP-server product that's live with PayPal. The case study we published at launch showed 91% fewer compile errors, 65% lower token consumption, 89% retrieval relevance, and 2x integration speed when an agent had deterministic API context, versus an agent flying blind on the same spec. The number that surprised me wasn't 91%. It was how many of those compile errors traced back not to the agent, but to the spec itself.
After enough of those debugging sessions, I started keeping a list. Ten dimensions that, in my experience, separate an OpenAPI spec an agent can integrate cleanly from one that quietly wastes a week of someone's life.
Today I shipped a free tool that scores the first five of them: A[P]Iaudit. Paste an OpenAPI spec URL, get a 0–100 readiness score and a ranked list of fixes. No signup, no paywall. This post is the opinionated bit: the framework underneath the score, and an honest accounting of where the tool ends and the backlog begins.
"AI-agent-ready" is a real shift in who's reading your spec
Before LLMs, an OpenAPI spec had two readers: humans browsing docs, and the SDK generator turning the spec into Python or Java clients. Both are forgiving in different ways. A human can read between the lines. A code generator only cares about syntactic correctness; it doesn't care if your descriptions are useful.
An AI coding agent is a worse reader than both. It's a generator and an interpreter at once, and it can't ask follow-up questions when something is ambiguous. If your description says "Returns user data," it'll generate code that returns user data and then guess what's in the response. If your error model is inconsistent across endpoints, it'll pick one shape and apply it everywhere. If two endpoints both call something id but mean different things, it won't notice.
"AI-agent-ready" is the property of a spec where an agent doesn't need to guess. It's a tighter version of "well-designed." All the things you already know good API design looks like (consistency, examples, predictable patterns) matter more, because the reader is less forgiving.
The encouraging part: this is mostly a documentation and discipline problem, not a redesign problem. Almost every fix is additive. You're not breaking your API. You're making it legible.
The 10 dimensions
These are the dimensions I think about, in roughly the order I'd ship fixes for them. The first five are what A[P]Iaudit scores today. The other five are on the roadmap, and I'll explain why I held them back.
| Dimension | Status | How the post says it is scored |
|---|---|---|
| 1 Description coverage and quality | Live in v1 | Split: deterministic coverage count, plus LLM-graded quality on a 0–3 rubric |
| 2 Example coverage | Live in v1 | Deterministic. Three coverage percentages, counted by the edge function, not the LLM |
| 3 Naming consistency | Live in v1 | LLM-graded. Known to be wrong on heterogeneous specs |
| 4 Auth clarity | Live in v1 | Not stated in the post |
| 5 Error model | Live in v1 | LLM-graded in part. Depends on cross-endpoint reasoning, which the per-operation prompts handle awkwardly |
| 6 Pagination and filter pattern uniformity | Roadmap, v0.2 | Not scored. Harder to score from the spec alone; needs several endpoints read together |
| 7 Idempotency signals | Roadmap, v0.2 | Not scored. Prompt design isn’t there yet |
| 8 Discoverability artifacts | Roadmap, v0.3 | Not scored. Not OpenAPI fields; a separate fetch |
| 9 Hallucination-prone fields | Roadmap, v0.3 | Not scored. Needs a grader that knows which kinds of fields get hallucinated |
| 10 One more, TBD | Roadmap, v0.3 | Not picked. Deprecation hygiene, versioning clarity, or webhook/callback documentation |
Live in A[P]Iaudit v1
1. Description coverage and quality. What percentage of your operations have a non-empty summary or description, and are those descriptions actually useful? The dimension splits coverage (a deterministic count) from quality (LLM-graded against a 0–3 rubric where 0 is "missing," 2 is "describes inputs, outputs, and at least one error case," and 3 is "explains when to use, parameter semantics, side effects, idempotency"). An agent generating an integration relies on these descriptions to choose between similar endpoints. "Returns user data" is a quality-0 description even though coverage counts it as present.
2. Example coverage. Three coverages, rolled together: request bodies with examples, 2xx response schemas with examples, and complex parameters with examples. Examples are the single highest-signal artifact for an agent. A schema with required fields tells the agent what must be present. An example tells the agent what those fields typically look like, which is the difference between a request that passes validation and one that gets a 400 because the agent invented a plausible-looking enum value that doesn't exist.
3. Naming consistency. Four sub-criteria: case style (snake_case, camelCase, kebab-case, or mixed, which is the red flag), verb-noun discipline (GET /users, not GET /getUsers), plural consistency (/users/{id}, not /user/{id}), and parameter naming uniformity (always limit/offset, not limit here and pageSize there). Mixed casing across endpoints is what bites agents hardest, because they'll pattern-match whatever they saw last and produce code that looks right but breaks at runtime.
4. Auth clarity. Is there one dominantly-used auth scheme, or are there five overlapping ones with unclear precedence? Specs that declare three security schemes and use a different one per endpoint group are technically valid OpenAPI, but they force the agent to make a separate auth-flow decision per call. The best specs declare one scheme at the document level and treat exceptions as exceptions, not as the norm.
5. Error model. Do your 4xx and 5xx responses share a schema? Is the error code field in the same place every time? Is there a documented code/type/message triple, or do some errors come back as { "error": "..." } and others as { "message": "...", "details": [...] }? Agents trying to write error-handling code for your API are doing pattern-matching at the response-schema level. An inconsistent error model means the agent's catch block handles one shape and silently breaks on the others.
On the roadmap (coming in v0.2 and v0.3)
These five are explicitly not in the tool yet. I'd rather ship five dimensions I trust than ten I'm not sure about.
6. Pagination and filter pattern uniformity. Cursor vs. page-number vs. offset, named consistently or mixed, and whether filters use a uniform query-parameter convention. This one is in the backlog because it's harder to score from the spec alone. You often need to look at multiple endpoints together and infer the pattern, which is a different prompt shape than the per-operation grading I'm doing today.
7. Idempotency signals. Are POST endpoints that should be idempotent calling out an idempotency-key header? Are PUTs actually idempotent? This is a known weakness for agents: they retry on network errors and sometimes create duplicate resources because the spec didn't tell them retries were safe.
8. Discoverability artifacts. Does the API have an llms.txt, a machine-readable changelog, an MCP server? These aren't OpenAPI fields (they're a separate fetch), but they materially affect whether an agent can pick up which API to use in the first place. (This is the "GEO is the new SEO" angle I wrote about a few months back.)
9. Hallucination-prone fields. Free-text fields where an enum would do. Magic strings buried in descriptions. Fields whose semantics are documented in prose but not in the schema. These are the places where an agent will confidently invent a value that looks right. Scoring this one well requires the LLM grader to know what kinds of fields tend to get hallucinated, which is genuinely harder than the structural checks above.
10. One more, TBD. I'm intentionally leaving a slot open. Three candidates are circling: deprecation hygiene, versioning clarity, and webhook/callback documentation. I'd rather pick the right one after running the v0.2 tool against a few more real specs than commit to one today.
A note on the score: why I think it's honest, and where it isn't
The thing that makes me nervous about every "AI evaluates X" tool is the same thing: who graded the grader? If a model is scoring your spec and you have no way to check the model's work, the score is a vibe. The PM in me sees this as the eval problem: model-graded metrics need human-graded ground truth, or they're decorative.
What gets counted, and what gets graded
A[P]Iaudit's v1 splits each dimension into a deterministic part and a graded part wherever possible. Coverage percentages (how many operations have descriptions, how many requests have examples) are counted by the edge function, not the LLM. The LLM only grades the quality of what's present, and only on a sampled subset of operations (up to 30, deterministically chosen by spec hash so the score is reproducible). The overall 0–100 score is a transparent weighted average of five 0–3 dimension scores, not a black-box number.
Where I expect it to be wrong
Three places, and I'd rather write them down here than have you find them yourself:
- Description quality on edge cases. A description like "Idempotent; safe to retry within 24 hours" is a quality-3 description in two sentences. The grader sometimes marks these as terse. I have a hand-graded calibration set of five reference specs being built (Stripe, GitHub, Twilio, one mid-quality customer spec, one intentionally-bad spec), and the calibration table will publish alongside the tool, showing exactly where the LLM and I disagreed, with the disagreement explained openly.
- Naming consistency on heterogeneous specs. When a spec has clearly different generations of endpoints stitched together (a common reality), the grader sometimes flags this as "mixed" when a more useful answer would be "modern endpoints are consistent, legacy aren't." That's a prompt problem on my end, not a model problem. I'll fix it in v0.2.
- Anything that requires cross-endpoint reasoning. Pagination patterns and error-model consistency partly depend on looking across operations. The current per-operation grading prompts handle this awkwardly. Some of why dimensions 6 and 7 are on the backlog is that the prompt design isn't there yet.
If you run the tool and disagree with a score, that's a signal I want. Tell me the spec and the dimension. The honest version of an "AI tool" is one that publishes its disagreements.
What to fix first, if your score isn't where you want it
Most of the specs I've audited so far fall into one of three failure modes, and the fix priority is different for each. Here's the playbook the tool tries to surface, in plain English.
If your description coverage is below 60%, fix that before anything else. Coverage is the cheapest dimension to move and the highest-leverage. An undescribed operation is invisible to an agent in any practical sense. It'll skip the operation or just guess. A one-sentence description is dramatically better than nothing. You don't need quality-3 prose on every endpoint; you need something on every endpoint.
If example coverage is your weak point, prioritise request bodies first, response bodies second, parameters third. Agents writing integration code spend most of their tokens constructing valid requests. A single realistic example on a request body removes more guesswork than ten extra sentences of description. Auto-generated example values from your schema (the kind tooling can produce) are better than nothing, but a hand-written example with realistic IDs and enum values is materially better. That's where the agent picks up the semantics of your fields, not just their types.
If naming consistency is dragging the score, this is usually a versioning problem in disguise. You can't snap your whole API to a new naming convention in a sprint. But you can document the inconsistency, pick the modern convention as canonical, and start enforcing it on new endpoints. The score will lag. The next agent integrating against your new endpoints will have a much easier time anyway.
The tool ranks the fix for you and names a specific operation when it can. That's the "top fix" callout on each dimension card.
What this tool is, and what it isn't
A[P]Iaudit isn't a replacement for an API linter. Spectral and Redocly already do excellent structural linting, and you should be running one in CI. This is the layer above: an opinionated take on what AI-agent-readiness means specifically. The bar I'm trying to hit: a PM or staff engineer can paste a spec in a meeting, look at the score, and know what to ship next.
It's also not a Context Plugins replacement. Context Plugins is the enterprise version of this conversation: deterministic API context delivered as an MCP server, with the case study numbers from the top of this post. If you're trying to make your API integration-ready for production agent workflows, that's the conversation to have. A[P]Iaudit is the free, public, opinionated front door.
If you run it on a spec you think is excellent, or one where the grader is clearly wrong, I'd like to know. There's an email-collect on the result page, and you can find me at ahshaikh.com or on LinkedIn. Source is at github.com/ahhshaikh/api-agent-score. Issues and PRs welcome.
Five dimensions today, ten by the end of the quarter. The part I care about most is the calibration table, which will tell you exactly where I, the model and the spec disagreed.
