# Eval Design for PMs Who Aren't Engineers

> If your team ships LLM features you need a rubric, not pytest. The simplest version of eval design I have found that survives a real spec review.

Author: Abdul Hannan Shaikh, senior technical PM at APIMatic.
Published: 22 August 2025  
Topic: AI product management  
Reading time: 5 min read  
Canonical: https://soch.ahshaikh.com/blog/eval-design-for-pms-who-arent-engineers/

---

Most PMs working on AI features stop one step short of the thing that would actually make them senior in the AI lane. They don't write evals. Not the engineer-grade kind, with YAML and CI. The PM-grade kind: a rubric, five test cases, and a one-page table that says "v2 is better than v1 on the things we care about."

Why do they skip it? I think it's the cultural framing. The tools that own the category (Braintrust, Promptfoo, DeepEval) are engineer-shaped: SDKs, configs, repos. So PMs decide evals are an engineer thing. They aren't.

## What an eval actually is

A rubric first, meaning one sentence that defines what "good" means for this feature. Then a fixed set of inputs you run the prompt against, and some way to score what comes back. That's the whole game. Everything else is plumbing.

If you can write a job description for an intern, you can write a rubric. "Friendly tone, under 100 words, never repeats a factual error the user states." The trick is writing it before you see any outputs. Read the outputs first and then ask yourself honestly: would you have written the same rubric an hour earlier? Post-hoc rationalization is the most common eval failure I've seen.

## The five-test-case minimum

Five test cases, picked deliberately, is where you start seeing real signal. One is a demo. Three is a vibe check. Pick the five to span the surface:

- One easy case the prompt should obviously handle
- One typical case from real user data, if you have it
- One edge case at the boundary of the prompt's intended scope
- One adversarial case: a prompt-injection attempt, or a factually wrong premise the model shouldn't accept
- One you genuinely don't know how the prompt will handle

The last one earns its place more often than you'd expect. If you can pass all five and your rubric is honest, you've done more eval work than 80% of teams shipping AI features.

## Scoring without writing code

Scoring five outputs against three criteria doesn't need an LLM-as-judge framework. It needs a table. Open a spreadsheet. Criteria across the columns, test cases down the rows. Fill in 1–5 by hand, one line of note per cell. The whole thing takes 20 minutes.

Running that same eval against v2, v3 and v4 without re-grading by hand is a different problem. That's the moment for an LLM judge or a real tool. Don't start there. The spreadsheet is what teaches you what your rubric is actually saying. A tool only runs it faster.

## The output is a one-page table

When you bring this to a spec review, the artifact is the table. Not the prompts. Not the architecture diagram. The table. "Here are the five cases, here's the rubric, here are v1's scores, here are v2's scores, here's the per-row delta." Engineering will respect that more than any prose argument about why v2 is better. Designers will use it as an anchor for their own critique. Your skip-level will read it in 30 seconds and say "ship v2."

| Test case | Friendly tone | Under 100 words | Never repeats a factual error the user states | Delta |
| --- | --- | --- | --- | --- |
| Easythe prompt should obviously handle it | v1 · v21-5 + one line | v1 · v21-5 + one line | v1 · v21-5 + one line |   |
| Typicalfrom real user data, if you have it | v1 · v21-5 + one line | v1 · v21-5 + one line | v1 · v21-5 + one line |   |
| Edgeat the boundary of the prompt’s intended scope | v1 · v21-5 + one line | v1 · v21-5 + one line | v1 · v21-5 + one line |   |
| Adversariala prompt-injection attempt, or a factually wrong premise | v1 · v21-5 + one line | v1 · v21-5 + one line | v1 · v21-5 + one line |   |
| Unknownyou genuinely don’t know how the prompt will handle it | v1 · v21-5 + one line | v1 · v21-5 + one line | v1 · v21-5 + one line |   |

*The whole artifact. Rubric across the top, the five cases down the side, a 1-5 score and one line of note in every cell, and a per-row delta on the right. The rubric shown here is the example one from earlier in this post, friendly tone, under 100 words, never repeats a factual error the user states; swap in your own three. It is blank because the scores are the twenty minutes of work only you can do, and because the test set has to be locked before the prompt changes.*

It works because the table moves the conversation from opinion to evidence. "v2 feels better" gets rejected by half the room. "v2 scores higher on the format criterion but worse on the brevity criterion" is a real conversation worth having.

## Where the rubber meets the road

A few habits separate the PMs who do this from the ones who say they do.

**Lock the test set before you change the prompt.** If the test set drifts to match the new prompt, you're running a demo again. Same five inputs, after every change.

**Document where the judge was wrong, whether the judge was you or a model.** Every eval has cases where the score is misleading and most of the room can feel it before anyone says so. Write those down while you can still remember why. The known failure modes are the part future-you will need and the part you won't recall unaided.

**Treat the rubric as a living artifact.** Your first version is wrong in ways you can't see yet. Update it as you learn what "good" actually means for the feature. The rubric is a forcing function for clarifying product intent, which is most of why it's worth a PM's afternoon.

If your team is shipping AI features and nobody owns the rubric, the PM should.