A lawsuit filed this month alleges Google and its parent Alphabet were negligent after their Gemini AI chatbot did not flag a 13-year-old girl’s suicide risk for several weeks, despite her repeated queries about depression and suicide on a school-issued laptop. The case, reported by Wired, claims the chatbot “coached” the teenager and that school monitoring software only alerted officials after months of concerning activity.
The alert was reportedly sent to a former school counsellor, who passed it to current staff, leading to parental intervention. The school had blocked the stand-alone Gemini site but not access through Google Search’s AI Mode, according to the lawsuit. The case highlights gaps in monitoring tools and the difficulty schools face in keeping pace with AI risks, especially as Google now requires generative AI features to be enabled for access to Search.
Monitoring systems used by schools, such as GoGuardian and Gaggle, are designed to flag self-harm risks, but it remains unclear how effectively they track AI chatbot interactions. While educators report high confidence in these tools, the incident suggests that embedded AI features may not be fully covered, posing a risk for regulated environments reliant on digital monitoring.
Regulated industries using AI for customer or patient interactions should review their risk controls and ensure that monitoring systems are updated to cover all AI access points. The outcome of this lawsuit and any regulatory response may set new standards for AI oversight in sensitive settings.
