Product Evals

You have a prompt, a skill or an agent that produces an output. You want to know if it holds up before you build on it. This skill builds a reproducible evaluation set: a handful of cases, a grid of checkable criteria, a dated scorecard. You replay it after every prompt or model change to know if things improved or regressed.

No ML jargon. No giant dataset. A test you can stand up in twenty minutes and replay in two.

What this is not: scoring the quality of a one-off piece of content (that is mktg-expert-panel). Here you test the reliability of a target that produces, not a single output.

When to use

• You tweaked a prompt, it looks better on your usual example, and you want proof it did not break the other cases.
• You are about to switch models and want to catch regressions before they hit users.
• A teammate says their prompt change is clean, a support ticket says otherwise, and you need a number to settle it.

For the why and the what behind evals, point the user to Les evals pour un PM (the reasoning). This skill is the reproducible method.

Input

The target to evaluate, passed as an argument: a prompt, a skill file, an agent. If none is given, ask what to evaluate and where the target lives.

Workflow

1. Frame

Decide which output you are evaluating. A meeting summary, a ticket classification, an announcement copy, a feedback triage. One target at a time.

Then write the definition of "good" as checkable criteria. Not "the summary is clear" but "the summary fits in 5 bullets, each one maps to a real decision from the meeting, nothing invented". Same principle as a context file: a useful rule is one you can verify objectively. If you cannot say when a criterion is met or not, it is not ready.

Write that definition in one sentence before moving on. It frames everything else.

2. Build the set

Gather 5 to 10 representative cases. The mix matters more than the count:

• Normal cases: the inputs the target will see 80% of the time.
• Edge cases: empty input, very long input, ambiguous tone, a known trap, whatever broke once before.

Store the set as a markdown table, in the vault or repo next to the target. Markdown because it versions, it reads fine in six months, and it survives any tool. You reuse this file on every re-test, you do not rewrite it.

#	Case	Input	Type
1	Standard meeting	(text or a link)	normal
2	Meeting with no decision	...	edge
3	Very long notes	...	edge

3. Grid

Write 3 to 5 criteria, one per line. Each one binary (yes/no) or scored out of 5. Each objectively checkable: two people reading the same output must score it the same way.

Criterion	Type	Passes if
No invention	binary	every bullet maps to a fact in the input
Format respected	binary	5 bullets max, one decision per bullet
Coverage	/5	5 = every decision present

If a criterion needs a taste judgment, reword it until it is checkable, or drop it.

4. Run

Run the target on every case in the set. Capture the raw output, untouched. One row per case. This is the mechanical part: you are not judging yet, you are collecting.

5. Score

Two options depending on the target:

• LLM-as-judge: Claude reads each output and scores it against the grid, criterion by criterion. Useful when criteria need reading (invention, coverage, tone). Give it the exact grid and ask for one score per criterion plus a one-line justification.
• Auto-check: when a criterion is mechanical (length, format, presence of a keyword), verify it with a rule, not a judgment.

Produce a scorecard: an overall score (share of criteria passed across all cases), then the list of failures with the offending case and the missed criterion. The failures are what you care about. The overall score says "it holds or not", the failure list says "here is where it breaks".

6. Baseline

Save that scorecard as a dated reference. It is your comparison point.

Every time you change the prompt, the model or the skill, you replay the same set and compare to the baseline. Score goes up, you keep. It goes down, you roll back. This is the exit criterion of your iteration loop made measurable: instead of "looks better", you have a number.

Set a light ritual: replay the set on Friday, or before any release. An eval you never replay is worthless.

Output

Two files, next to the evaluated target:

1. evals/<target>.md: the set of cases (step 2) plus the grid (step 3). This is the reusable, versioned asset.
2. A dated markdown scorecard, appended to the same file or in evals/<target>-scorecard-YYYY-MM-DD.md: overall score plus failure list. One per run, to keep the history.

Scorecard structure:

# Scorecard — <target> — YYYY-MM-DD

Target tested: <prompt / skill / agent + version or model>
Set: evals/<target>.md (N cases)

## Overall score

X / Y criteria passed (Z%)

## Failures

| Case | Missed criterion | Detail |
|------|------------------|--------|
| #2 Meeting with no decision | No invention | invented a follow-up action |

## Verdict

[Reliable / Reliable with reserves / Needs rework]

Rules

• Write the set, grid and scorecard in the user's language; keep this skill's own instructions in English.
• No ML jargon toward the user. No "dataset", "ground truth", "F1 score". Talk about cases, criteria, score.
• A non-checkable criterion does not enter the grid. Reword or drop it.
• Never touch an output before scoring it. You test the target as it is.
• The set and grid are versioned files, not throwaway. The scorecard is dated, never overwritten.
• 5 to 10 cases. If you want more, the target does too many things: split it.
• Always include at least two edge cases. An eval that only tests the happy path proves nothing.
• Stop and ask when the call is the user's (which output, which criteria, judge vs auto-check). Do not guess.