Learnings from 'Creating a culture of experimentation'

As the emphasis on collaboration and innovation within our organizations is increasing, more engineering leaders are turning toward implementing a culture of experimentation.

But how can this be cultivated? What are the challenges? And what is the real impact of experimentation on our teams and our products?

In Creating a culture of experimentation, we explored these questions and more, highlighting the impact that a culture of learning can have in the tech industry, and on the engineers that fuel it.

Episode 1: The challenges of introducing product experimentation

In this article, product managers, Monira Rhaimi and Oli Gibson, share their learnings from attempting to introduce a culture of experimentation in their org, and discuss the unexpected challenges they faced. They understand that organizations can be concerned about the potential roadblocks that could occur in integrating experimentation, and have provided a toolkit to challenge these worries.

They tell the story of their company stopping the development of a multi-million-pound product that had been taking a ‘build-first’ approach. This led them both to look at bringing in experimentation, but they swiftly realized that there were many bumps in the road; particularly involving stakeholders. They write ‘We were taking the lowest-risk approach for building products, and using the smallest amount of money as efficiently as possible. Ironically, this made it harder to get investment.’ They were competing against well-established and financially-sound products, as well as a budgeting process that was not designed for incremental funding.

The struggles of operating within an organization that didn’t have a data-driven culture as it’s foundation meant that the experiments’ results were not being met with the reaction from stakeholders that Monira and Oli were hoping for. This article highlights the importance of establishing a learning culture before embarking on experimentation, and the challenges that no one tells you about.

Episode 2: Establishing experimentation as a core part of your project workflow

This conversation revolved around introducing experimentation into an organization, with our panelists – Sha Ma (VP Software Engineering at GitHub), Justina Nguyen (Developer Evangelist at Optimizely) and Robyn Rap (Data Science Manager at Indeed) – and moderator –  Leemay Nassery (Engineering Manager at Spotify) discussing their own experiences and learnings from making this cultural shift.

From choosing experimentation platforms, to surprising test results, our panelists personally reflected on what experimentation meant to their teams, and how they overcame the challenges they faced. During this discussion, our panelists explored:

• The importance of qualitative data and talking to your users
• How to choose metrics for AB tests
• How to overcome the challenges of building a scientific culture in your org 
• Tactics for gaining buy-in
• The different types of experimentation methods
• Why celebrating the results of each experiment is important

Episode 3: Building stronger teams with AB testing

Jason van der Merwe sets us up with the notion that product decision-making always involves assumptions regarding the users’ needs. He explains that a culture of experimentation acknowledges this, but that it also ‘encourages learning about whether those assumptions are correct or not before substantial investment is given to the project’

He highlights what can go wrong when you develop without testing, using Snapchat’s 2018 redesign as an example, and that these mistakes are usually centered around the fear of failure and hurting metrics. When you convert this fear of failure to an opportunity for learning, you encourage risk-taking with the ‘safety net’ of AB testing, therefore driving innovation and psychological safety within your teams.

Jason continues to explore a culture of experimentation through increased collaboration within teams, and the promotion of leadership and entrepreneurship. He explains that ‘it is the entire team’s responsibility to ship great solutions to users’, and so by allowing the whole team to be ‘involved in ideating and brainstorming a solution from the beginning’ you acknowledge that ‘everyone on the team has good ideas’, resulting in stronger teams and better product decisions.

Episode 4: Adopting an experimentation philosophy

In the final article of the series, Justina Nguyen presents a practical guide to implementing an optimization methodology in your org whilst also discussing both the positive impact it can have on your teams and the challenges that it raises.

Justina outlines the entire process, covering how mapping can serve as an operational checklist; the importance of connecting your experimentation to company goals; tips on running ideation sessions; how to prioritize your team’s ideas; the do’s and don’ts of execution; and the impact of analyzing your results. Justina then dives into potential bottlenecks, and why measuring and benchmarking your team’s performance is fundamental in keeping your engineers focused and driving execution.

Adopting an experimentation philosophy takes time and dedication, but Justina’s article discusses the need-to-knows in order to ease the process.

A final takeaway

Cultures of learning and experimentation enable better product development – but for many within an organization, a move towards this culture can be one filled with fear.

Justina writes that ‘creating a process is simple, but mobilizing people to follow that process can be challenging, [but by] encouraging and rewarding a test-and-learn mindset’ an optimization methodology can be driven into an org and make an impact.