When Wrong Looks Right
AI-generated safety info can sound confident, plausible, polished – but can be very wrong in ways it’s difficult to spot…
When Wrong Looks Right
One of the most deceptive things about AI is when it gets something wrong, it rarely looks broken. In fact, when it fails, it can fail beautifully.
If a calculator gives you an error message, you know there’s a problem. AI can give you something that’s superbly structured, detailed, confident, and professional. It can cite legislation, identify hazards, suggest controls, and use exactly the sort of language you’d expect to find in a safety document.
And still be wrong.
AI companies often describe these mistakes as hallucinations. In practice, that can mean inventing a fact, misrepresenting guidance, citing legislation that doesn’t apply or even exist, missing an important qualification or giving you an answer that’s broadly correct but wrong in precisely the way that matters. But outright errors aren’t the only problem.
AI systems are designed to be helpful. They tend to accept the premise of the question they’re given and try to provide an answer. If you ask for ten hazards, they’ll generally try to give you ten hazards. If you ask for controls, they’ll give you controls. And if you ask for a risk assessment, they can produce something that looks remarkably like one.
The document may look complete. But the assessment is the judgement behind it – not the document itself.
And that’s particularly important in safety, because a polished document can create a sense of reassurance. It has headings. It has hazards. It has controls. It might even have risk scores and a neat red, amber and green table.
It looks like the work has been done. But AI doesn’t know what you forgot to tell it. It doesn’t know that something unusual about your location, contributor, equipment or activity can change the situation completely. And it doesn’t necessarily know when it should stop answering and start asking questions.
So don’t judge AI-generated safety information by how professional it sounds. You need to judge it by a much harder question: what is this answer based on – and what might be missing?
Because with AI, a convincing answer isn’t necessarily evidence of a good assessment. Sometimes it’s simply evidence that AI is very good at writing convincing answers. And there’s an even more subtle problem. Sometimes there’s nothing obviously wrong with the answer at all.
The problem is the question it was answering.
