Why we have to be extra careful about AI risk in government

After a week of stark warnings about AI, Ben Pirt, Principal Technologist at Made Tech, argues that public sector organisations need to separate what could go wrong from the impact if it does, and stay healthily sceptical as AI becomes routine.

Getting AI wrong can have a much greater impact in government than it would in many commercial settings. If Amazon gets an order wrong, it is frustrating but usually fixable. If incorrect information feeds into a government decision, the consequences can be much more serious.

At Made Tech, we work with public sector organisations on complex transformation programmes and use AI ourselves where it can help teams work more effectively. This is not an argument for government keeping the technology at arm’s length. It is about understanding what happens when something goes wrong in a setting where the consequences matter.

I used ChatGPT to help plan a holiday in Norway recently and it did a really good job. Had it made a mistake, I might have missed somewhere worth visiting or taken the wrong route. That would have been annoying, but ultimately not that important.

Compare that with a system producing incorrect information that informs a decision about somebody’s case. The technology may still have made an error, but the fallout can be life-changing and, in some circumstances, extremely difficult to put right.

What actually happens if it goes wrong?

There is a lot of discussion about AI risk at the moment, but treating it as one large category does not get us very far. When assessing AI, it is important to separate what could go wrong from the impact if it does.

From my point of view, you need to understand the different things that can happen when you use AI, work out which of those apply to the particular setting, consider what the impact would be if one of them played out and then decide what you are going to do to mitigate it.

Bias is a useful example. If I am using AI to analyse source code, bias may not be particularly significant. If I am using it to analyse transcripts from user research, it probably matters more because the model could influence how those findings are interpreted. If AI-generated information is then being presented to somebody making a decision about an individual case, the potential impact becomes much more significant.

The same thinking can be applied to other risks. The important point is that the consequences depend heavily on where and how the technology is being used. The more serious the potential impact, the more carefully organisations need to think about how the technology is used and what happens if it gets something wrong.

We already know how damaging it can be when a system's output gains more authority than it deserves. The Post Office scandal was not an AI failure, but it demonstrated what can happen when a system says something is true, people act on it and there is not enough challenge around that assumption.

That should make us careful with the language of moving fast and breaking things. There is good reason for government to experiment with new technology, but what exactly are we prepared to break?

Familiarity can make us too casual

Part of the difficulty is that AI has become normal remarkably quickly. It is increasingly built into search, browsers and workplace software, and people are getting real value from it.

That everyday experience influences how we think about the technology. If you regularly use AI to search for information, draft something or organise a holiday, it becomes very easy to think: it’s fine, I use this all the time.

I think that is where we have to be careful in the public sector. Familiarity with the technology does not remove the need to think about the particular setting in which it is being used.

It also does not mean every use of AI needs to be treated as though the consequences are enormous. The point is to understand the risks that genuinely apply and the potential impact in that situation, rather than either glossing over them or treating every application in exactly the same way.

When you actually work through what could happen, the conversation can change quite quickly. Something that initially seems like a harmless or obvious use of AI can look different once somebody asks what happens if the output is wrong and who is affected as a result.

Keep the ability to challenge it

Deskilling is another risk that I think deserves more attention.

AI can be hugely useful. In technology delivery it can help people work more quickly, understand unfamiliar material and get through tasks that previously took much longer. But there is a difference between using AI to support somebody’s expertise and gradually losing the expertise needed to judge its output.

People often say AI is only as good as the prompt you put into it. I would add that it is also only as useful as your ability to look at what comes back and know whether it is any good.

That ability matters even more where the impact of an error is high. If people are going to challenge AI-generated information, they need to retain the knowledge and confidence to recognise when something does not look right.

For me, that is why being healthily sceptical is a useful position. It does not mean being anti-AI, and it certainly does not mean ignoring the benefits. It means understanding where things can go wrong, taking the possible impact seriously and not accepting an answer simply because the technology presents it confidently.

Sign up for our session on 24 September at 14.10 here. Ben will be joined by Claire Potter, CDIO Data Science, AI Ethics and Assurance Lead at HMRC, Jenny Brooker, Chief Data Architect at GDS, Richard Baines, Deputy Director, Digital Delivery, Digital Data Technology & Security at DEFRA and Tom Collins, Head of Service Creation at DVLA to discuss ‘AI is the Tool. Your people are the key. Managing the Risks of Public-Sector Legacy Modernisation’