AI governance frameworks in high-risk industries may be eroding proactive human judgement, according to MIT Sloan Management Review. By making humans reactive to AI-generated alerts, rather than proactive in identifying threats, these frameworks risk undermining behaviours that have kept sectors like nuclear power and air traffic control safe.
High-reliability organisations have learned that safety depends not only on technology but also on distributing decision authority to those closest to the problem and treating anomalies as signals. Past disasters, such as Three Mile Island and the Columbia shuttle loss, have shown the dangers of suppressing front-line judgement and failing to prioritise alerts.
AI-enabled threat-detection systems now promise broad pattern recognition but remain prone to bias, drift and false positives. These can overwhelm staff or miss critical threats, as recent cyber attacks and healthcare breaches have shown. The article notes that current AI governance often mirrors high-reliability organising principles, but warns that over-reliance on AI alerts can create new risks.
For regulated firms, the key issue is whether governance frameworks support or erode the judgement of those closest to operational risk. Reviewing escalation channels and decision rights may help avoid blind spots and maintain reliability.
