How confident should we be in AI that handles money?

Test accuracy tells you how a model did on yesterday's data. When software decides whether a payment is fraudulent, we need a defensible claim about how often it will fail tomorrow.

When a machine-learning model approves a loan, flags a payment as fraudulent or decides that a login looks suspicious, it is making a decision that matters to someone's money. The usual way to describe such a model is with a single number: accuracy on a held-out test set. That number is useful during development. It is not, on its own, a claim anyone should rely on once the system is live.

Accuracy is a statement about the past

A test-set score describes how a model performed on data that was collected, cleaned and labelled before deployment. Real operation is different in at least three ways. The mix of inputs changes as customers, merchants and fraudsters change their behaviour. Rare but costly cases, which matter most, are by definition under-represented in any sample. And the test set was usually drawn from the same pipeline as the training data, so it shares the same blind spots.

Reliability engineering asks a sharper question: what is the probability that this system fails on the next demand, and how confident are we in that estimate? For a fraud model, a "demand" is a transaction it has to judge, and a failure might be letting a fraudulent payment through or blocking a legitimate one.

Evidence from operation

The most convincing evidence about how software behaves in operation is observing it in operation, under conditions that resemble real use. Suppose a system has been exercised on n realistic demands and has not failed once. What can we claim?

A simple Bayesian calculation gives a feel for the numbers. If we begin with no preference for any failure probability, and we observe n failure-free demands, then to be 99% confident that the probability of failure per demand is below one in a thousand, we need roughly 4,600 failure-free demands. To claim one in ten thousand with the same confidence, we need about ten times as many. The evidence required grows in proportion to the strength of the claim. There is no shortcut.

Why "conservative" matters

The uncomfortable part of any Bayesian argument is the prior: the beliefs we hold before seeing operational data. Choose a convenient prior and almost any conclusion becomes available. This is where conservative Bayesian inference, developed over many years in the software reliability community, is valuable.

Instead of committing to one prior, the assessor states only what they can defend, for example "we are 90% confident from development evidence that the failure probability is below one in a hundred". The method then finds the worst prior consistent with those statements and reports the confidence that still holds. The resulting claim is weaker than an optimistic analysis would give, but it is one that survives scrutiny, because it does not depend on assumptions nobody can justify.

What changes when the software learns

Applying these ideas to machine-learning software raises questions that classical reliability methods did not have to answer. A model's behaviour depends on its data, so evidence gathered under one distribution of inputs may say little about another. Models are retrained, which raises the question of how much old evidence still applies to a new version. And failures are often silent: a fraud model that misses a payment does not crash.

These are open research problems, and they are the reason reliability assessment of ML systems is an active field. The direction of the answers is already clear, though: claims should be tied to an explicit description of the operating conditions, evidence should be re-weighed when those conditions shift, and confidence should be stated alongside every estimate.

What this means for financial products

For a company building AI into financial services, the practical lessons are straightforward to state and demanding to follow.

Define failure precisely, per decision, before measuring anything. A missed fraud and a wrongly blocked payment are different failures with different costs.
Collect operational evidence deliberately, under conditions that match real use, including the rare and expensive cases.
State claims with their confidence, and make the assumptions behind them visible to the people who will rely on them.
Treat a retrained model as a new system until evidence shows otherwise.

None of this makes AI less useful in finance. It makes its usefulness something we can demonstrate rather than assert, which is the standard people are entitled to expect from software that handles their money.