As an engineering manager, how can you make sure you are using metrics effectively to help your team’s workflow and make problems and risks more visible?

As part of our series, ‘The engineering leader’s guide to data-driven leadership’, LeadDev brought together a group of engineering managers to discuss how they use metrics to support their daily planning, and how they can use data to run better standups.

Group discussion

The session kicked off with a discussion around what metrics folks are currently using, and their biggest challenges and worries around metrics. Most attendees agreed they use metrics to measure some part of the development cycle, including delivery lead time, build time, code review time, merge time, pull request throughput, and deployment frequency. A few said they also use data to gauge the way in which their teams are working, tracking velocity, current work in progress, blocked time, and burndown. Just one attendee said they use people metrics to measure satisfaction, engagement, happiness, and diversity within their team.

The most common concern was knowing when and how to use metrics. Folks agreed it’s not always clear what you should be measuring, so you need to understand exactly what you want to manage, and what matters to your team before you start capturing data (which is easier said than done). Inconsistent results were another worry, especially at the team level. When teams measure different things, comparing results isn’t straightforward – or necessarily always helpful.

How Code Climate uses metrics

Next, Khan Smith, VP of Product at Code Climate, gave a presentation on how they use data to improve management practices and run better standups. Looking at metrics from a people perspective, he shared how managers can plan and coach teams more effectively, with an eye towards psychological safety. Here are his key takeaways.

• Most engineering managers start their day by asking themselves four questions: Is there smoke coming from anywhere in my team? Is this really a fire? How big is this fire? And what is the root cause? But you can also ask a fifth helpful question: How can I use data to optimize this constant planning and risk cycle?  
• To get started, first identify the risk conditions of a potential problem, and how to define them. For example, you might define a long review as in the review stage for more than five days with more than two review cycles. Next, pinpoint the data you need access to. With the right data, you can find and triage risk quickly and more effectively without having to go into a bunch of different systems. Finally, assess the data by asking forensic questions. For a long review, you might ask: Is there a lot of back and forth on comments for a particular PR? How many people are participating in the discussion? Are they junior or senior? Once you understand what’s behind the activity, you can make your go-forward plan.
• Dig into the data before running standups so that you come prepped with insights about the team and the individuals. This way, you can save valuable time by getting straight to solving the right problems. You can call out blockages (which aren’t always reported), reassign workload if folks are overwhelmed, and fill in any knowledge gaps which you can see are leading to delays.
• Remember: Metrics allow you to uncover gaps, risks, and problems without constantly tapping your team on the shoulder to ask for updates. Psychological safety is a huge factor here; using data allows you to surgically help your team while empowering them by giving them space to do their own work.

Group discussion: Running better standups

After Khan’s presentation, the group came together again for a final discussion around how they run daily standups, and how metrics can help to surface more potential problems and trigger more valuable conversations.

Most of the group agreed the most common format for standups is asking these three questions: What have you done in the last 24 hours? What will you do in the next 24 hours? And what, if anything, is blocking your progress?

To prepare for standups, many folks recommended looking at hard metrics to help you understand the status of current items, and where things might need work. For example, you can check if anything is hung up in QA waiting on review; look at open PRs to see their status; review any outstanding tickets to make sure your team is being responsive; and check current progress compared to estimations.

One attendee shared a top tip: Try setting up Slack notifications for PRs that are open too long. Creating daily alerts for any request that is open for more than three days can help you to spot delays early on. By automatically tagging the folks responsible, you can improve accountability and help teams manage their workload. These notifications helped the attendee to reduce their slow PR count from 30 to 6 in 1 month!

Reflections

As the session came to a close, the group agreed that working with metrics isn’t always straightforward. We use data to create concrete pictures of complex situations, but knowing what to measure, and when, can be hard. But when used effectively, metrics can be a powerful tool for managers, helping you to see under the hood of all your team projects. By building data into your daily planning and standups, you can catch more risks before they turn into problems, you can trigger more productive conversations, and, perhaps most importantly, you can create a psychologically safe environment for your team.