This post, which covers the second half of our conversation with Patrick Kirwin, Silent Eight’s Head of Product Management, unpacks what makes an AI agent's decision defensible, the product design choices that keep humans genuinely in control, and where agentic systems are most likely to fail in a regulated environment.

What do regulators or model risk teams typically need to see to trust that AI is defensible?

If we're going to allow AI to make autonomous decisions on something as complex and important as financial crime risk, there needs to be a certain set of things in place to give people trust that we can let the system do that.

First is explainability. Specifically for an agent making decisions, that needs to come in the form of a human-readable outcome, a descriptive narrative in plain language that explains the rationale behind the decision. 

We're no longer accepting a risk score or a threshold-based probability with a list of contributing features. The outcome needs to be understandable not just by a data scientist who can read the code, but by anyone who picks it up.

Second is evidence of guardrails. The organisation needs to know under what circumstances the model can make a decision autonomously, and what's out of bounds. That matters because if you optimise purely for efficiency, the model will give you an answer on everything, even when it isn't confident, and that comes at the cost of getting it wrong. The model needs to be allowed to say “I don't know.”

Third is evidence of human-in-the-loop governance like quality assurance sampling, or controls over human approval of new model learnings. It's not enough for guardrails or human oversight to simply exist; there has to be evidence of them for that trust to be real.

What specific design features ensure human control over models?

I'll be a bit more product-specific to Silent Eight here, but it extrapolates in general terms too.

First, guardrails need to align to the customer's specific risk tolerance. No two institutions have the same risk appetite, so these rules should look different for every customer, and they need to be accessible to business users, not just data scientists who can read the code. In our products we use the concept of policies and codes, which let customers configure the application to meet their risk tolerance and visualise it.

Second is quality assurance. Our products allow decisions to be made autonomously without human oversight, essentially auto-closure of risk. But we back-test a proportion of those decisions against actual human review to make sure we're always in line, so it doesn't become a self-fulfilling prophecy where the same kind of decision keeps getting made whether it's right or not.

Both of those need to be evidenced too. If a policy or code changes, or a sampling decision is made, it's done by a human, and there's a track record of it.

What are agentic systems’ main weaknesses in regulated environments, and how can those weaknesses be guarded against?

With agentic systems, the risk is that decisions are being made at scale. If one human analyst makes a bad decision, the impact is fairly small. They probably only looked at ten cases that day, and once I spot the gap I can retrain that person and look back over what they worked on. 

Agentic systems, by contrast, are making decisions at speed and scale. Hundreds or thousands of cases a day, using the same criteria, across the entire spectrum, not just one person's queue. So if something goes wrong, the scale of the impact is much bigger. The same guardrails I talked about earlier still apply. I need to be doing sampling, I need human oversight into what the agent is doing, and when I catch an error I need to fix it quickly so it doesn't keep compounding.

The flip side is that this scale cuts both ways. If I find an error and retrain the model, I'm fixing it for every case that comes afterward, for the whole company, in one go. If it's a human, I might have to retrain ten, twenty, a hundred people, which takes a lot more time and steering to get quality back to where I want it. So the same speed and scale that makes errors dangerous also makes corrections far more efficient.

Looking out over a few years, which development are you personally most genuinely excited about?

If I talk about not even a few years out, but right now, what I'm genuinely excited about is truly embracing agentic workflows. This exists today and we use it today, but once the industry really pushes the limits of what can be done here, I think we're in for some exciting times.

It's not just about cutting costs or making operations more efficient, though that's a benefit. I think we're in a place where we could make a real dent in financial crime. Banks have always been in catch-up mode. We're always behind the criminals, more of a nuisance to them than a disruptor. I think the speed and scale we can bring with agentic research and investigations could actually be a disruptor in the financial crime space, a way to get ahead of criminals and operate at the same level, rather than always playing catch-up.

Have you changed any of your beliefs relating to AI and compliance over the years, or is there one you'd most like to retire?

I was in this camp at some point in my career where I didn't believe AI could play a significant role in risk identification or decisioning. Over a decade ago, I was working with someone on a data science team who was new to financial crime. I told him what risks we needed to identify, and he said “just give me all the data about your customers and transactions, and I'll tell you what's suspicious, we'll find risks you didn't know about.” I didn't believe that. I had all this career knowledge; there was no way he'd find something I didn't already know.

Looking back on it just two years later, he was absolutely right. There were knowledge gaps and blind spots we had, and technology is a tremendous way to fill those, highlighting risks we never thought of and finding patterns faster than ever. So the belief I'd like to retire is that doubting attitude, the hubris of thinking my own experience was the only reliable method. That experience is still valuable, but I've learned to let technology assist in ways that move us forward in detecting financial crime.