Adopting an experimentation philosophy

How can you cultivate a productive experimentation program that yields results?

Creating a unique and seamless digital experience is very challenging given the competition, yet optimizing and personalizing these experiences as a practice is often siloed, under-resourced, and as a result, underperforming. How can companies maximize their investments and take full advantage of the resources they already have to achieve better performance? 

What we’ll cover:  

• Remove bottlenecks and unify siloed & understaffed teams by formalizing an optimization methodology, empowering individuals and groups to take ownership to execute it;
• Holistically address the tooling requirements that each step of the optimization methodology presents, including research, program management, experience delivery, and analysis; 
• Foster latent creativity in your team by instituting a regular cadence of research (an intelligence reporting process) to ensure ongoing improvements with an optimization program; 
• Ensure focus by benchmarking your team’s output with operational metrics unique to an optimization program.

Empower teams by formalizing a methodology 

The foundational step towards maximizing the value of optimization is to map out the inputs, outputs, team involvement, and milestones that the work requires. This mapping will make it easy for various teams to understand the commitments they will need to make, and serves as an operational checklist that will streamline workflow, and align individuals and departments with each other. This methodology is universal in its core structure, but allows flexibility, and can be adapted to organizations of various sizes and structures. 

Preparation

When you build an optimization strategy for your company, you’ll first align your program’s testing efforts to your company goals. Without a strategic guide, you run the risk of building unfocused or inefficient tests that consume time and resources without producing results. To ensure that the product experiments you run are impactful, you’ll need to connect the experimentation program to key company metrics, and identify a core team to drive and implement the technology and program. 

Ideation

Ideation, the process of generating powerful hypotheses to drive experiments and campaigns, is one of the most important steps in experimentation. Without a structured framework supporting the creative stage of experimentation, it can be difficult to consistently generate impactful ideas. Before an ideation session, gather as much data as you can by analyzing product usage and talking to customers so that you can provide direction for your brainstorming session.

Planning

Now that you’ve had a few ideation sessions and multiple people from different teams have contributed exciting ideas, you’ll need to prioritize and plan to execute those ideas. To figure out which experiments and campaigns to run first, and which to place into your backlog, use a prioritization framework to evaluate them. Most organizations find it helpful to align experiment ideas with the product roadmap, and tactically outline steps for each experiment using a scalable framework. This ensures that experiments have company-wide support from a resourcing perspective, and always stay top-of-mind. 

Execution

Once you’ve decided on the priority order of your experiments, the next step is to actually build and implement them. A rigorous QA process ensures that your experiment looks and works the way you want it to before you show it to visitors. Once you’ve finished building your experiment, be diligent and take extra time to verify your setup so you can trust your results, and more importantly, deliver frictionless user experiences. 

Analysis

Your experiment results – whether winning, losing, or inconclusive – are an incredibly valuable resource. The data on your results page helps you learn about your visitors, make data-driven business decisions, and feed the iterative cycle of your experimentation program. After running an experiment, make sure you take the time to interpret your results, and most importantly, share them with your company.

Foster creativity with an intelligence report

Research suggests that one of the most pressing bottlenecks in an optimization methodology is the ability to consistently generate and execute a volume of quality optimization hypotheses.

Generating hypotheses is one of the most cost-effective bottlenecks to remove, because unlike a lack of development resources, absence of the proper tools, or cultural reluctance to commit to optimization, teams and individuals have relatively direct control of idea generation for experimentation. One root cause of this issue is that the real output of optimization is, very often, what traditionally has been the purview of creative specialists: graphic designers, UX specialists, copywriters, or creative directors. Experimentation is, in many ways, a discipline with a ‘creative’ output. Yet, the vast majority of optimization programs are led by individuals or groups with ‘non-creative’ backgrounds, like performance marketers, analysts, project managers, merchandisers, etc. While design and UI are very often involved, the teams that drive optimization are often business units or engineers: marketing, ecommerce, product, engineering, etc.

Drive focus, drive execution: benchmark operational metrics 

Even for experienced, well-resourced teams, dynamics unique to optimization present consistent challenges. Most organizations meet these challenges in an unstructured way, relying on the individuals closest to the work to address them as best they can. In this report, we’ll address the challenges of experiment velocity, experiment quality, program efficiency, and program agility. For leaders trying to maintain operational excellence, measuring and benchmarking the team’s performance in meeting those challenges ensures that focus is kept on execution. This executional focus allows executives and program managers to recognize and address potential issues earlier, while serving as a reliable leading indicator for overall program performance, and therefore serving as excellent quarterly goals. 

Each of these leading indicators has a qualitative connection to the optimization process and a quantifiable variable. By tracking, reporting, and socializing these variables in a dashboard or similar snapshot communication method, the team will be able to gauge progress and make course corrections to strategy, process, and resourcing. 

Conclusion 

Building an experimentation culture takes time because creating a process is simple, but mobilizing people to follow that process can be challenging. When experimentation is first introduced at a company, organizations are usually invigorated at the onset but can sometimes feel overwhelmed by the process. A productive experimentation program requires discipline in order to yield results that can positively impact the user experience, and the key to driving that program is encouraging and rewarding a test-and-learn mindset.