Operations · 8 min read · Aug 12, 2026
Agree the pass mark before you build.
Evaluation is the least glamorous part of an AI build and the one that most reliably decides whether it reaches production. A pilot without an agreed pass mark cannot conclude — it can only continue, and eventually be abandoned for reasons nobody can quite articulate.
Why pilots run forever
A pilot with no defined success criterion has no state in which it is finished. Output is reviewed, opinions are formed, some are positive and some are not, and the decision defaults to whoever is most senior or most tired.
The fix is unglamorous and it has to happen before the build: write down what the system must get right, on which cases, at what rate, agreed by the people who will judge it. That single artefact is what allows a pilot to conclude in either direction — which is the point, because a pilot that cannot fail was never an evaluation.
If nobody can state the number that would make this a success, the project has no way to end. Agree it while it is still cheap to disagree.
Where test cases come from
Invented test cases produce misleading confidence, because people invent tidy examples. Real evaluation sets come from real history, and a good one has four ingredients:
- Ordinary cases. The routine bulk. Establishes baseline competence and is the easiest to gather.
- Historical exceptions. Cases that previously required a human decision — this is where systems actually fail.
- Known-hard cases. The ones experienced staff can name from memory. Ask them; they will produce a list in minutes.
- Adversarial cases. Malformed input, contradictory data, missing fields, and content designed to mislead.
The second category carries most of the value and is the most commonly missing. A system evaluated only on routine cases will pass evaluation and fail deployment, because production is where the exceptions live. If you take one thing from this: go and get the exceptions.
Measure what failure costs, not just how often
A single accuracy percentage hides the thing that matters. Two systems at 92% can be very differently deployable, because the shape of the 8% differs.
| Question | Why it matters more than accuracy |
|---|---|
| Which way does it fail? | Missing something is not the same as inventing something |
| Does it know it is unsure? | A flagged low-confidence case is a handled case |
| What does one failure cost? | Decides how much review the process needs |
| Is failure detectable downstream? | Silent errors are the expensive kind |
The second row is the most useful property to design for. A system that is right 85% of the time and reliably flags its uncertainty is more deployable than one that is right 95% of the time with uniform confidence — because the first can be operated with a review queue and the second cannot be operated at all without reviewing everything.
The human baseline nobody measures
Evaluations are usually run against an implicit assumption that the current manual process is correct. It is not, and comparing against a perfect standard that does not exist sets an unreachable bar.
Have people redo a sample of historical cases without seeing the original answers. You will find disagreement between them, and disagreement between the same person on different days. That number is the honest benchmark, and it is frequently lower than everyone expects. It also identifies genuinely ambiguous cases — which should be routed to a human by design rather than counted as model failures.
Evaluation is not a phase
The most common structural mistake is treating evaluation as a gate before launch rather than a permanent fixture. Model providers update models. Behaviour changes on upgrade, including on changes described as improvements. A system validated against the previous version has not been validated.
Keep the evaluation set runnable on demand and run it after every model change, every prompt change and every retrieval change, plus on a schedule. It should be the cheapest thing in the project to execute — if running it is a two-day manual exercise, it will not be run, and the system will drift unobserved.
The same set doubles as a regression suite. Fixing one failure mode frequently introduces another, and without a stored set of cases that were previously passing, nobody notices until it reaches a customer.
Dealing with this in your own group?
We answer scoping questions before there's a contract in sight — including the ones about cost and data handling.