Commercials · 7 min read · Aug 12, 2026
Measure the process, not the pilot.
Ask why an AI project cannot prove its value and the answer is almost never that the value was absent. It is that nobody wrote down what things looked like before. This is a measurement design problem, and it is solvable — but only in advance.
The baseline is the whole game
The single most common failure is starting without a measured baseline. Once a system is live, the pre-system state becomes a matter of recollection, and recollection is generous in whichever direction the person answering prefers.
Capture it before anything is deployed, and capture more than the headline: how long the process takes end to end, how many items go through it, the error and rework rate, how much of it is exception handling, and how the volume varies across a month. That last one prevents a comparison against an unrepresentative week.
A baseline captured after go-live is not a baseline. It is a reconstruction, and it will be argued with by whoever does not like the conclusion.
Time saved is not money saved
The standard calculation multiplies hours saved by an hourly rate and produces a large number that finance correctly refuses to believe. The reason it is wrong is straightforward: unless the hours convert into either fewer people or more output, nothing has been saved. It has been redistributed.
Distributed savings are real but they are not bankable. Twenty people saving fifteen minutes a day is a genuine improvement in working life and it is not five headcount. Claiming it as five headcount is how AI programmes lose credibility with the finance function permanently.
What is bankable is specific:
- Capacity absorbed. Volume grew and headcount did not — the clearest and most defensible form.
- Cost avoided. A hire that was planned, budgeted and did not happen.
- External spend removed. An outsourced service or licence cancelled.
- Error cost reduced. Fewer credit notes, penalties, reworks — measurable if you were counting before.
- Cycle time. Faster close, faster quote, faster response, where speed has a demonstrable commercial effect.
Count the whole cost, including the permanent part
The cost side is usually understated by omitting everything after launch. A defensible figure includes build, integration, running cost across inference, infrastructure and integration — and supervision.
Supervision is the line that gets treated as temporary and is not. Someone reviews the exception queue, samples auto-approved output, and re-tests after model changes for as long as the system runs. Business cases that model oversight as a first-year transition cost systematically overstate payback.
Measure the process, not the tool
"Did the AI work?" is unanswerable and invites anecdote. "Did invoice processing get faster, cheaper or more accurate?" is answerable, and it correctly captures the cases where the model performs well and the process does not improve — because the bottleneck was approval, or because exception handling now takes longer than the manual path it replaced.
That distinction matters because process-level measurement is the only kind that catches partial automation making things worse. A system that handles 70% of cases and hands over a badly-framed 30% can reduce total throughput while showing excellent model accuracy.
A measurement design that survives review
| When | What to do |
|---|---|
| Before build | Measure the baseline across a full cycle, not a week |
| At scoping | Agree the metric and the pass mark in writing |
| At go-live | Run in parallel where feasible — same inputs, both paths |
| Ongoing | Track the process metric, not model accuracy |
| Quarterly | Re-measure; behaviour drifts after model upgrades |
The second row is the one that decides whether the exercise concludes. Agreeing the pass mark before build removes the possibility of the target moving to fit the result, in either direction — and it is the same discipline that separates a pilot which concludes from one that runs indefinitely without a verdict.
Parallel running is underrated and cheaper than it sounds. Same inputs through both the old path and the new one for a bounded period gives you a direct comparison on identical work, which no amount of before-and-after reasoning can match. It also surfaces disagreements while there is still a manual path to fall back on.
Dealing with this in your own group?
We answer scoping questions before there's a contract in sight — including the ones about cost and data handling.