The idea that a human should always be involved in AI-assisted decisions has become something close to received wisdom. But one of the more striking moments in a recent webinar on AI and public sector fraud was the suggestion that if that human is not genuinely engaged and empowered, they may offer very little protection at all.
The discussion touched on automation bias, the limits of human oversight, and what it actually means to keep a person meaningfully in control when the machine is fast, eloquent, and apparently authoritative.
Akber Datoo, founder of D2 Legal Technologies and a professor of law, technology and AI at the University of Surrey, was direct on this point. He described human accountability in AI-assisted processes as something that cannot be ceremonial. A human approver, he said, needs to be trained, empowered, and genuinely able to disagree with the machine.
The problem is automation bias - the tendency to accept the machine's output without meaningful challenge. Akber described it as people quietening the rightful voice inside themselves that is, in fact, the whole point of having a human in the loop. The machine produces a well-structured, confident-sounding result, and the temptation is simply to go along with it.
Akber used a vivid analogy to describe the drift that can set in when organisations become accustomed to AI tools. "We think we're taking the dog out for a walk," he said, "but it's starting to pull us along. The dog's taking us for a walk very quickly with these tools if we're not careful."
Rob Savage, VP for Public Sector at NUIX, echoed this concern from a technical perspective. He noted that one of the biggest risks with large language models is the eloquence with which they respond. Outputs are well-structured, well-referenced, and often compelling in tone. Over time, he said, users can begin to forget that they are dealing with a probabilistic model that is providing the most likely response - not necessarily a correct or complete one.
Rob described this as "reliance drift" - a gradual process in which people become more dependent on the system and lose sight of what it is actually doing and what its limitations are.
One audience member, asked a question that got to the heart of why AI decisions make people uncomfortable even when human decisions are no more reliable: given the same data, the same rules, and the same methodology, what fundamentally makes a human decision different from an AI decision?
Akber's response was that the difference is institutional and moral, not necessarily one of accuracy. With a human decision, he said, responsibility has a clear location. The reasoning can be requested, the judgement can be challenged, and there are established mechanisms for holding someone accountable if they fail to do what they are supposed to do.
With AI, he argued, imperfection is made to look objective. A machine output carries what he described as a false aura of neutrality. And when things go wrong, responsibility becomes blurred - spread between the user, the vendor, the model, and the organisation. "The key issue is not human good, AI bad," Akber said. "It's coming back to explainability, challengeability, and we just know how to keep humans accountable in the way society has grown up."
Nick Jennings, Head of the Hertfordshire Shared Anti-Fraud Service, brought a practitioner's perspective to the same question. He noted that AI will sometimes simply decline to make a determination, saying - in effect - that this is not a decision for it to make. Other times it will produce a conclusion that is difficult to understand from a human perspective. Neither response is wrong, but both illustrate why the human role cannot be passive.
Rob raised the point that AI can operate continuously in a way that humans cannot, which has real implications for disclosure and investigation quality. He described a scenario involving disclosure in a prosecution case - where a person reviewing tens of thousands of documents over a prolonged period may eventually become unreliable through fatigue alone. AI, he suggested, offers a way to address that - but only if the human oversight that accompanies it is substantive rather than nominal.
Akber, reflecting on the legal context, reinforced this with a clear principle: AI is not evidence. At best, it is a route to evidence. The person presenting findings in a formal setting has to understand how those findings were reached, what the limitations of the process were, and how it can be challenged.
This post is based on a webinar on "AI and fraud prevention in the public sector", featuring speakers from the Public Sector Fraud Authority, the Hertfordshire Shared Anti-Fraud Service, NUIX, and D2 Legal Technologies. Listen to the whole thing for free here >> https://register.govnet.co.uk/webinar-ai-vs-fraud