Measuring and improving the efficiency of software delivery

Highly effective and productive engineering teams are constantly exploring opportunities to drive efficiencies in software delivery.

We are living in a digital world where customers are expecting faster (and more) value delivery from their service providers and they expect this value to be delivered with high quality. 

Driving efficiencies will not only help with faster value delivery but also offers the benefits of staying nimble, being competitive, and helps with increasing top-line and bottom-line growth of the company. 

At Indeed, we have been running velocity programs for over the last two years, which has led us to embrace efficient techniques and processes for delivering more value to our customers. 

Why do you need to drive for efficiency in software delivery?

Driving efficiency really matters. It can help with the speed of learning and innovation which is essential for an organization to harbor an effective time-to-market strategy and to stay competitive in the marketplace. Efficiency also improves developer morale and productivity, and helps to improve the unit cost economics of value delivery. If an engineering organization of 100 developers can drive a 10% improvement in delivery efficiency, it can result in increasing the throughput by 10 more developers!

How can you measure software delivery efficiency? 

Before an organization dives into the different strategies for measuring efficiency, it should focus on continuous and incremental improvement over time with a focus towards good change – essentially embracing Kaizen principles for driving change. There are different ways to measure the efficiency of software delivery, including: 

• Throughput. The total number of changes that can be completed and deployed to production over a given time interval.
• Scrum velocity. The average number of story points per sprint.
• Delivery lead time (DLT). The average time to deliver a unit of work to production. 
• Deployment frequency. How frequently teams deploy code. 
• WIP limits. The number of active units of work at different phases of the delivery life cycle. 

It is also important to ensure that there is a balance between driving improvement in efficiency metrics and maintaining a high quality that can be measured by metrics, like:

• Major bug escapes. The number of major bugs impacting customers in production.
• Change fail percentage. What percentage of production deploys result in a failure.
• Mean time to restore (MTTR). How long it takes to restore a service failure. 

Every organization is different and will have to identify the right efficiency metric that is right for it; working backward to identify the current state of that metric and make continuous improvement on it by measuring and optimizing across the organization. 

What did we choose as our metric for measuring efficiency at Indeed?

 Out of the metrics discussed above, we chose DLT as a measure of our success. Some of the motivations for picking this over others included:

• It has a direct correlation to faster value delivery;
• It improves delivery throughput;
• It provides optics into the key bottlenecks of the delivery life cycle;
• It’s easy to measure using a ticket tracking system like Jira.

How did we develop our metrics at Indeed? 

We were using Jira to track the teams’ work and extracted data from Jira for every transition of a ticket to a different state. We extracted this data into our internal data storage, and built tools and dashboards to provide insights and recommendations to different teams on opportunities for improving delivery efficiencies. We also integrated the data storage with other systems that were involved in the software delivery, like the CI/CD system, deployment system, monitoring system, etc. – and provided a holistic view on the opportunities for driving efficiencies. We’ve also sent periodic emails to different teams to show their progress towards their expected improvements.

Challenges we faced when driving change 

We faced a few challenges in aligning the organization with this focus. We primarily focused our strategy around addressing developer concerns of gaming the stats, Goodhart’s law, optimizing the metric, and decoupling performance evaluation while measuring these metrics. We emphasized leveraging these metrics for continuous learning and improvement to be centered around teams (and against individual developers) for driving efficiencies. We empowered the teams to forget the metric and live the values centered around agility, quality, and reliability. We were careful to not evaluate people on these metrics as that can drive bad behavior; we focused on celebrating the team wins when they made significant improvements.

Rollout of the change

To roll this out at scale, we emphasized the importance of change management across the organization and used a seven-point approach (Define, Communicate, Align, Develop, Train, Incentivize, and Measure) for driving the change. This rollout included:

• Defining our goal of reducing DLT by 50% in a calendar year;
• Communicating our vision and strategy to senior leadership, stakeholders, and teams;
• Sharing our vision and strategy in newsletters and company all hands to be very explicit about our goals and intent; 
• Aligning team goals with DLT improvement goals to meet objectives;
• Developing tools and dashboards and training teams to use them to improve their engineering velocity;
• Changing engineering rubrics to clarify the value of driving these improvements and incentivized the team for the right behaviors;
• Constantly measuring and reporting progress;
• Recalibrating based on our learnings.

Conclusion

There is great value in continuously exploring opportunities for driving efficiencies in software delivery. You and your organization should identify the right process (centered around Kaizen) for driving this change and understand what it means for your domain and business. Find out the right metrics that are relevant to your teams based on their maturity, and more importantly, ask your teams how they feel about improving delivery efficiencies and develop a strategy to drive the change. Some of the steps we’ve taken above can be applied in rolling out the change across your organization, and we hope they help in your journey to improving efficiency.