Register or log in to access this video

New York • September 8 & 9, 2027
Loved LDX3 New York? Pre-sale tickets for 2027 are now available.
When a production AI agent hallucinated, the true crisis wasn’t technical—it was organizational. Discover the battle-tested governance playbook we built to map accountability and safely deploy high-stakes automation anywhere.
Every engineering leader has shipped something that kept them up at night. In most industries, a bad release means a degraded UI or a temporary drop in conversion metrics. In healthcare, it means something else entirely.
This case study details the rollout of AI agents into clinical workflows handling prior authorizations, discharge summaries, and medication triage across more than 2 million patient interactions annually. Built on AWS Agent Core with Agent Squad orchestration across specialised clinical agents, the architecture was sound — the ownership model was not.
Three weeks into production, a discharge-summary agent hallucinated a medication continuation instruction. No patient was harmed, but the fallout was immediate: clinicians lost confidence, overrides skyrocketed, and our engineering teams entered a finger-pointing loop. Platform engineering, ML, and clinical operations had all assumed someone else held final escalation authority. Nobody did.
This talk shares the operating playbook we built to pull our AI system out of crisis and back into production safely — and more importantly, to make sure we never ended up in that room again. We’ll cover how we drew strict ownership boundaries for agentic systems where no single service owns the full outcome, how we engineered human-in-the-loop patterns that preserve genuine accountability without becoming rubber stamps, and how we rebuilt clinical trust through review structures that gave humans meaningful control rather than the illusion of it.
The governance framework that came out of this incident has since been adopted by two teams outside healthcare entirely — because it turns out, regulated AI is just a more honest version of the AI governance problem every engineering leader is quietly avoiding.
You do not need to work in healthcare for this talk to matter- this is a blueprint for any leader deploying high-stakes automation.
Key takeaways
- A reusable playbook for defining escalation authority before an incident forces the conversation.
- A decision-ownership model for agentic systems where multiple specialised agents share a workflow but no single service owns the final outcome.
- A governance structure for regulated AI that gets systems out of endless pilot phases and back into production with organisational sign-off.
- A trust-rebuild framework for restoring stakeholder confidence after a high-stakes AI incident — applicable far beyond healthcare.
- A human-in-the-loop design pattern that preserves genuine clinical accountability without collapsing into rubber-stamp override behaviour at scale.