AI in product management
AI evals for product managers: how to define “good” for an AI feature
Updated
When your product includes an AI feature, “does it work?” stops having a simple answer. The same input can produce different outputs, and quality is partly a matter of judgement. Evals are how teams turn that judgement into something they can measure and improve. The product manager is often the best person to lead them, because evals are really a definition of what good looks like for your users.
Why evals are a product job
Off-the-shelf metrics such as “helpfulness” or “toxicity” sound rigorous but rarely match what goes wrong for your users. Hamel Husain and Shreya Shankar, who teach eval methods to product teams, warn against dashboards full of metrics that have no link to real user problems. Their process starts with a person who knows what good looks like reading real output. In many teams that person is the product manager.
Step 1: Read real output before measuring anything
- Collect about 100 real interactions. Sample them at random from production or a beta. Not your favourite demo cases.
- Judge each one: pass or fail. Write a short, free-form note on what is wrong with each failure. Husain and Shankar call this open coding.
- Group the notes into failure modes. Cluster similar problems into a short list, ideally fewer than ten. AI can suggest groupings, but review them yourself.
- Count them. How often does each failure happen? That ranking is your priority list.
Failure mode Count Example Cites a policy that does not exist 14 #023 refund policy Ignores the user's stated constraint 9 #041 budget under 500 Correct but far too long 6 #077 Wrong tone for an upset customer 4 #012
Step 2: Turn failure modes into checks
- Objective failures get code. Missing fields, broken formatting, a link that 404s and responses over a length limit can be checked by simple, reliable code.
- Subjective failures get an AI judge. For tone, relevance or faithfulness to a source, a second model grades the output against a clear rubric. This is called LLM-as-a-judge.
- Use pass/fail, not 1–5 scores. Binary judgements are easier to agree on and to act on. Keep the written notes; they carry the nuance and make good examples for the judge.
Step 3: Check the judge against yourself
An AI judge is only useful if it agrees with the person who defines quality. Label a set of examples yourself, run the judge on them, and compare. Husain and Shankar recommend measuring two numbers instead of overall accuracy: how often the judge catches real failures, and how often it correctly passes good output. Accuracy alone hides problems when most outputs are good.
Then decide which mistake costs you more. For a medical or financial assistant, letting a bad answer through is worse than flagging a good one. For a writing helper, the opposite may be true. Tune the judge for the error you can least afford.
How evals fit with experiments
Evals and experiments answer different questions. Evals ask “is the output good?” and run before release and on every change, like tests. Experiments ask “does this change user behaviour the way we hoped?” and run with real users. You need both: an AI feature can pass every eval and still fail to move the outcome you built it for.
- Use your eval pass rate as a guardrail in experiments: the new prompt or model must not lower it. See success metrics and guardrails.
- Keep the primary metric about user outcomes, such as task completion or time saved, not about model output.
- Re-run evals whenever you change the prompt, the model or the data the feature draws on. Small changes cause surprising regressions.
Start this week
You do not need an eval platform to begin. Export 100 recent interactions to a spreadsheet, add a pass/fail column and a notes column, and work through them. By the end you will know more about your AI feature’s real failures than any dashboard could tell you.
Sources
Keep your evidence and decisions linked
Groundwork takes your team from customer insights to evidence-based requirements. Free to start, with your whole team.
Get started