Becoming AI engineers

How our team went from building simple AI features to multi-agent systems at the edge of what’s possible, and everything we learned along the way.

Every product today would be built differently if starting from

scratch with AI. The opportunity for applying AI in existing domains is huge, and we’re all being asked to AI-ify our products overnight. Doing this in a way that is truly transformative demands fundamental changes in our approach to building software.

I’ve spent the last year leading this transformation at incident.io, taking us from simple AI features like generating summaries to fully-automated investigation systems that dig through code, observability data, and historic incidents to identify root causes.

Getting there required us to reinvent how we build software. After all…

How do you test a system that never gives the same response twice? How do you make your product feel snappy when every request takes seconds? We’ve had to build totally new foundations: eval frameworks to grade AI behavior, tools that introspect multi-step AI interactions, and patterns like speculative execution to hit our latency targets.

Beyond the technical, these tools fundamentally affect how teams work. Even our most experienced engineers found themselves doubting their abilities, watching features that worked perfectly one day fail mysteriously the next. We ended up building tools that could prove our progress: tracking every AI interaction, grading responses, and giving engineers concrete evidence they’re moving forward, even when it doesn’t feel like it.

In this talk, I’ll share what we have learned from working with AI. I’ll show you our debugging tools, training harnesses, and architectural patterns. You’ll see real examples of how we build and test AI features, and how we’ve built resilience in our team to handle constant experimentation and frequent missteps building systems no one has built before.

AI isn’t replacing engineers but it is changing how we build software. Like devops bringing operations into the lives of software engineers, this is a revolution in how we work: challenging to master, but entirely learnable. Come see how we did it.

Key takeaways

• An overview of the challenges faced by engineering teams who begin working with AI
• Blueprint for applying ‘evals’ to your codebase just as you would normal test frameworks
• Optimization tricks and patterns like speculative execution that get the best performance from AI
• See real-world examples of tools that help grade, debug and track AI interactions in production
• Discuss strategies to evolve engineering culture and build resilience when working with emerging technologies like AI