How to develop engineering metrics with people, process, and tools in mind

Building and using engineering metrics will take time and practice, but by fostering a positive, data-driven culture, you will increase your team’s engagement, remove impediments, and see higher productivity.

The idea behind measuring a team’s operation is not new, but engineering metrics continue to be met with resistance.  And from my experience of talking to other engineering leaders, the practice goes sideways when it reduces metrics to vanity figures, without added context.  I understand how developers have come to resent this practice, but I promise you there is a better way. I want to encourage leaders to focus on the importance of their role in supporting, growing, and ensuring their team members are happy and engaged at work. With this in mind, engineering metrics can drive objective conversations with the team on what issues they face and how to best improve process, tooling, and general practices.

In this article, I will dive into how to develop your own engineering metrics and derive value from them. The goal is to use metrics as a tool to answer questions about continuing to improve your operation for your team members to thrive. 

Let’s work through a few questions to help you identify which metrics to collect, and how they help to identify issues. 

What matters most to your team right now?

Startups need to focus on quick experimentation to validate the best market fit. Mature products need to focus on resilience and pay off their technical debt. You may have a need to balance both. Gathering metrics can help you raise visibility on how much your team can invest to accomplish feature development as well as quality on a regular basis. 

It’s important to note that what we are looking at measuring here is the team operation not the product metrics. Product metrics help you identify how your product is being used and it can be a great tool for understanding what needs to be built next. Here, we are only talking about engineering team operation.

What has an impact on your team’s ability to focus on what matters most?

If your goal is to improve the operation of your team, this should be one of the key questions you spend time on. You may be surprised to hear the answers from your team but please listen carefully. There may be issues such as a lack of clarity around priority; the number of meetings on an individual’s calendar; or a lack of proper tooling which translates into more time spent working around the limitations rather than on implementing features. You may not make immediate changes right away from these answers, but by keeping those impacting factors in mind you can start to develop metrics that help you identify the true cost of these issues, and hopefully help you and your team make changes for the better.

Outline your guiding principles

To build a healthy and positive culture for your team, you need to be clear on your guiding principles. Doing so, as cheesy as it may sound, will have a big impact on how you lay out the incentives for your team that will transfer to your culture. Metrics can define how your team works to achieve their goals, and your team should care about those numbers; otherwise, there isn’t any real value in implementing them.

On my team, we emphasize iteration and collaboration as two of the guiding principles that we believe contribute to our success. In other words, we succeed by breaking work down to the smallest deliverable, and we value team effort over individual heroics. With these two principles in mind, I keep metrics at the team level, and I measure indicators that show how well we are breaking things down.

We hold ourselves to delivering with quality, so with each deliverable, we look to cut scope and we incrementally deliver work.

I can’t stress how important it is for you as a leader to keep these principles in mind. A healthy, balanced, and collaborative team can achieve amazing things. 

Outline your team’s workflow

Understanding the steps each unit of work needs to go through can help you see where in the process these units may get stuck, or take too long. For distributed teams, I have seen that often the review cycle is where a good amount of time is spent. If the team is distributed across a wide range of time zones, you will likely notice your cycle time (mean time to deliver) go up. These are all important and valid constraints, but knowing that this is where a pull request (PR) will take a few days before moving to the next step, can help the team identify ways to optimize. You may want to pair developers in close time zones, or you may need to document your code review guidelines so every developer is clear on what is expected of them. You will get the most value from working with your team on how best to improve.

What are you measuring?

I measure throughput: the weekly count of how many PRs the team have merged to production. I break this metric down by investment categories, so we can see how much we invested that week on chores, features, security issues, and bugs. But you can choose to define any number of categories. 

The balance of these investment categories is important because you are asking your team to support and be on-call for their services. If uptime is a priority, you need to invest in more than feature development. I recommend pairing throughput with quality-based metrics as it helps the team discuss what ratio may be needed to invest in engineering initiatives. Engineering initiatives (chores, bugs, refactors, etc.) should help improve the speed and quality of developing features; so for us, these investments go hand in hand.

I find that throughput helps the team focus on the incentive to break down the work into the smallest deliverable. It takes time to build this practice. Be patient with your team, and remember that this is one of the most valuable incentives. 

Ship small, ship fast, learn, repeat. 

Small PRs mean quicker reviews, less risk, and the ability to mitigate an issue quickly if it were to happen in production.

If you are concerned about gaming this metric, please don’t be. I cringe when I hear leaders talk about their team members as though they are not capable individuals who love their work. I believe everyone on my team is capable and as such, they are best to decide how much further they should break down their work. A bug may be a one-line fix, but a feature will likely require more. That’s OK. The value of these metrics is less in the raw number, and more in the trend. 

It will take time for these numbers to normalize, but there is no need to stress the team about this. I encourage the team to focus less on the number itself, and more on what it is telling us. What’s the story behind these numbers? You need to practice debugging your metrics, look at what is contributing to the number, and try to understand it. This is also important because data integrity is going to be the number one issue you face to begin with, so make sure you check your source.

How are you measuring this?

I am able to get the data for throughput from git. We use predefined labels that we added across all our repositories to allow us to filter on the four types we care most about. We aggregate this total weekly.

This is a team-based metric, which is important to us. As I mentioned when we were discussing guiding principles, we value teamwork and recognize that individuals need to help each other to deliver on the team’s commitments. This means that at times, one person might be doing more code reviews, and another might be on-call and investigating an issue in order for the rest of the team to focus on the next milestone of the features we have prioritized. The team’s work is what we represent, talk about, and value. This is how we collaborate and iterate.

Key takeaways

Build a metrics-driven culture. I review metrics with my team regularly, and we share why numbers are up or down. We look to understand and answer questions. You need to do this regularly for metrics to be effective in driving continuous improvements within your org.

Metrics should never be punitive. Metrics should help your team have a healthy and objective conversation about what is getting in their way. Foster a safe space for your metric reviews. Put emphasis on learning and improvement. 

It’s more about the trendline. I tend to focus less on the raw numbers and more on the trends. Are the metrics cyclical? Why is that? Maybe the way your Sprints or releases are set up force the team to pause or push harder towards a deadline. At times that might be needed, but I have found that success in a continuous development model is to see consistency week-to-week. Help your team find a sustainable pace so they can avoid burnout. 

Implement a dashboard. It is tempting to try and identify the one measure that will reflect how the team is doing, but from my experience, it is really valuable to have multiple measures that complement each other. I balance throughput with cycle time, and I balance both with quality metrics. I also add a soft metric using a pulse survey to gauge the sentiment of the team week-to-week, and to collect feedback. 

Keep it simple and iterate. A complex measurement is not likely to help you identify issues quickly. Keep iterating. 

Be transparent. You will not get the most value out of metrics if you do not share them with your team, and encourage them to have an objective dialogue on what these numbers represent. 

I love metrics and I have found them to be essential in managing and leading engineering teams. Building and using engineering metrics will take time and practice, but by fostering a positive, data-driven culture, you will increase your team’s engagement, remove impediments, and see higher productivity.

If you would like to find out more about the specific metrics I implemented in my team, you can read about them in more detail here.