Building stronger teams with AB testing

Just a few years ago, AB testing wasn’t prevalent at Strava. But as we built and scaled a culture of experimentation, we saw the immediate impact of better product decision making.

We also learned that product experimentation was helping us build stronger, more inclusive, collaborative, and creative product teams. In this article, I will dive into three learnings on AB testing, and how they have shaped the way our teams operate at Strava.

Encouraging risk-taking 

By providing a safety net that promotes learning, a culture of experimentation can help product teams feel encouraged to take risks. When planning and designing a new feature or product, good product teams utilize user research and analytics data to inform their decisions. But any plan is also born out of a set of assumptions, which we like to disguise as “intuition”. Sometimes our intuition is spot on, and sometimes it’s way off. As Daniel Kahneman, author of Thinking Fast and Slow, said, ‘The mere fact that you have an idea and nothing else comes to mind and you feel a great deal of confidence – absolutely does not guarantee accuracy.’

This is why it’s important to differentiate our assumptions from what we actually know. A culture of experimentation acknowledges that we make assumptions about our users and their needs. It encourages learning about whether those assumptions are correct or not before a substantial investment is given to the project. The purpose of AB testing isn’t just about whether some change is “good or bad”. It’s about learning how changes impact the user experience.

So how does this encourage risk taking? By identifying what we don’t know about our users, we can set up tests that allow us to learn these answers. We can try riskier things because the goal is to increase our knowledge of our user, versus a goal of simply increasing our key business metrics. These learnings can be applied to future decision making.

In 2018, Snapchat shipped a redesign of their app that not only wasn’t received well, but it drastically hurt key business metrics, including active users. One of their mistakes was assuming too much about their users and not testing along the way. Evan Spiegel said: ‘We rushed our redesign, solving one problem but creating many others … Unfortunately, we didn’t give ourselves enough time to continue iterating and testing the redesign with a smaller percentage of our community. As a result, we had to continue our iterations after we launched, causing a lot of frustration for our community.’ TechCrunch writer John Constine points out ‘Spiegel always went on his gut rather than relying on user data like Facebook. Aging further and further away from his core audience, he misread what teens cared about.’

Businesses and teams make these mistakes all the time; Snapchat’s was just very visible and a good lesson for us all. Spiegel and Snapchat took a big risk, but they didn’t set out with the goal of learning. Their mistake wasn’t attempting the new redesign; it was not creating the frameworks to learn from their risks before committing the whole business to that direction. Innovation doesn’t come without risk-taking. However, if the sole goal of a team is to move a metric, the fear of failure and hurting those metrics will paralyze them and hinder innovation. But by replacing the fear of failure with the opportunity to learn, AB testing provides a safety net that encourages teams to take risks.

Increasing collaboration

A culture of experimentation can help product teams by increasing collaboration and the creativity of the team, resulting in better ideas. Many teams operate with a waterfall development process, even if they use a form of Agile. In waterfall development, a user or business problem is handed to a product manager, who ideates a solution, then hands that solution off to be designed. The final designs are then handed to engineers for implementation. Waterfall development requires the feature or project to be fully defined so it can be passed between functions effortlessly.

The one issue with this process is that it removes the ability for the non-ideators to contribute to the formation of the solution. It limits who gets to creatively add to the solution. Teams are typically constructed with roles that are responsible for a specific scope of work. A product manager decides what we build, designers decide how it looks and acts, engineers decide how we build it, and analysts decide how we measure what we build. But at the end of the day, it is the entire team’s responsibility to ship great solutions to users, and that responsibility should be shared among the team. We should all care about what we ship to users. But if you’re at the end of that assembly line, your participation is limited.

So what if we redefined those roles earlier? A product manager should be responsible for defining the user or business problems we are trying to solve and for facilitating a solution; designers help us empathize with users; engineers materialize the decided solution; and analysts help us understand what we learned from the solution we shipped. The entire team is involved in ideating and brainstorming a solution from the beginning.

A culture of experimentation acknowledges that everyone on the team has good ideas that should be pursued. On my team, ideas are born out of discussions on how to solve specific problems. By the time we hit implementation, it’s not obvious where the original idea came from (though it’s almost always not from me!).

This idea is nice and makes sense, but it’s not unique to experimentation, right? Any good product building process will include the whole team in ideation. I agree, but a culture of experimentation helps foster this type of collaborative environment because our goal is learning. It acknowledges that intuition isn’t always correct, and introduces a willingness to try things we might not have tried before. It’s so easy for one or two people to become the gatekeepers for ideas, but intuition only gets us so far. Why wouldn’t we harness the creativity and problem solving of the entire team? Experimentation gives us more reasons to try new things and fewer reasons to say no.

One of the ways we do this on my team at Strava is through a day called “Experiments Day”. Every quarter, the team gets to spend the whole day building and shipping a product experience as an AB test. There are only three rules: 

1. Your test must ship to 10% of users; 
2. Your test must seek to improve an area of the user experience the team is currently focused on; 
3. Your test must be small enough to complete during Experiments Day. 

If your idea passes those three rules, you’re allowed to ship – no questions asked. The Follow Back button on athletes’ profiles on Strava is a result of Experiments Day and it was one of the most successful tests we ran.

Promoting leadership and entrepreneurship

The last way that a culture of experimentation builds stronger teams is by promoting ownership, buy-in, and entrepreneurship on the team. A culture of experimentation that encourages risk-taking, acknowledges the prevalence of assumptions, and encourages ideas from everyone, is a culture where psychological safety can grow and flourish. Psychological safety is ‘a sense of confidence that the team will not embarrass, reject or punish someone for speaking up… the shared belief held by members of a team that the team is safe for interpersonal risk-taking.’ (Amy Edmondson, Harvard Business School.) Experimentation, where failure is just a learning opportunity, is a safe place that allows people to step up and be leaders. It helps every person on the team develop empathy for the user by encouraging everyone to participate in solving user problems. 

When ideas for experimentation come from everyone, an experimentation culture creates a culture of leadership. All individuals are encouraged and empowered to speak up with their ideas. On my team, every experiment has an “owner”. This person is almost never the product manager, but is one of the individuals deeply involved with that particular experiment.  Experiment owners are expected to see the idea through the experiment life cycle: from idea, to design, to shipping, to understanding the impact, and to facilitating a decision on next steps. We often joke on the team that we have one lead product manager, and several mini PMs. This way of working results in a team that is highly empathetic to our end users.

A team of action-oriented, entrepreneurial teammates is a happy, productive, and creative team. Morale is higher when everyone is involved in decision making and solution crafting. And best of all, our team output is simply at a higher quality because we are harnessing the best of each other.

Conclusion

I hope I’ve been able to show you how a culture of experimentation has value beyond making better product decisions. Even if you’re not on a product team, a culture of experimentation can be extended to any situation where you are working with other people. I want to leave you with four principles that we try to embody on my team at Strava. If you implement these on your team, I am confident you’ll see your team grow stronger, and perform at a higher level.

• Everyone should get the chance to be creative and to be a leader;
• Be slow to judge someone else’s idea;
• Be quick to collaborate with your teammates;
• Take risks and focus on your learnings.