Assume it will. Design the blast radius.

Reversibility is the design axis

Accuracy is the wrong first question, because a 99% accurate agent posting journals is more dangerous than a 90% accurate agent drafting a summary. What matters is what the wrong 1% can reach. Sort every proposed agent action onto this ladder before arguing about models:

Most programmes that get into trouble did not fail technically. They put an agent two rungs higher than the surrounding process could check, and the first visible error cost the whole initiative its mandate.

Confidence scores are not a control

A model's own confidence is a useful signal and a poor guardrail: it is generated by the same process that produced the error you are trying to catch. Treat it as a routing input, not as permission.

What works is a hold-back queue. Anything ambiguous is not guessed and not silently dropped — it goes to a person, with the candidate answers and the reason it was held. Ontilus uses exactly this in item-name normalisation: several signals have to agree — normalised name, unit and pack size, supplier, observed price band — and anything short of agreement waits for a human decision rather than merging two genuinely different products.

A system that holds 8% of cases for review and is right about the other 92% is usable. A system that guesses on all 100% and is right about 96% is not, because nobody can tell which four are wrong.

Every action needs a run record

When something is wrong three weeks later, the only question that matters is what the system saw and why it decided. That requires recording, per action: the inputs it read, the output it produced, the model and prompt version, the rule that fired, who approved it, and when. Without it, an agent error is not investigable, and an uninvestigable error on a regulated process is a finding.

This also settles a question that comes up in every scoping call: can an agent run overnight without supervision? Actions on the bottom two rungs, yes. Anything higher runs unattended only if the run record and the alerting are good enough that a person can reconstruct the night's decisions in the morning.

Human gates that do not become rubber stamps

An approval step decays. Present a person with 300 correct items and they will approve the 301st without reading it — which is why volume through a gate matters as much as the gate existing.

• Route by exception, not by everything. A gate that fires on 3% of cases gets read. One that fires on all of them gets clicked.
• Sample the auto-approved. Pull a random slice of what the system decided alone and check it on a schedule. This is the only mechanism that catches quiet drift.
• Watch the hold-back rate, not just the error rate. A sudden drop usually means an upstream change, not a smarter model.
• Re-test after every model change. Model behaviour moves between versions; a prompt that was reliable in one release is an assumption in the next.

Rolling out so the first error is survivable

The sequence we use exists for this reason: a single pilot entity taken all the way through to reconciled numbers before anything goes group-wide, and nothing widens until the pilot's output agrees with that entity's own close reports. It looks slower written down and it is faster in practice, because an error found in one entity is a fix, and the same error found across twelve is a credibility problem.

The same logic drives sequencing by checkability rather than by size of the prize, which is covered in where back-office agents pay back.

The contractual half

Architecture decides how bad an error is. The contract decides who fixes it. Establish before signature who is on the hook when an agent misbehaves in month nine, what the response time is, whether fixes count as maintenance or as change requests, and how a model provider's own change — a deprecation, a price move, a behaviour shift — is handled. Those belong on the list in the questions to ask an AI vendor before you sign.