Using metrics to communicate the state of your engineering org

As an engineering VP, it can be difficult to keep track of your teams and effectively report their progress to the executive team and board.

Using metrics can help to monitor and communicate the state of your engineering org. But it can be difficult to know where to start. What are the right ways to measure success? Which metrics should you use when reporting to a management board or executive level? In what ways are other VPs using metrics that they find helpful or harmful?

As part of our series, ‘The engineering leader’s guide to data-driven leadership’, LeadDev brought together a group of engineering VPs to discuss how they are currently using metrics, and share their advice for getting it right.

The session kicked off with a group discussion about how attendees are currently using data to measure the success of their teams. Many shared that they track OKRs linked to their product vision or business objectives. A couple agreed that they monitor sprint story points, comparing their committed and completed points. Just one attendee said they use metrics to measure developer happiness, which they believe can be a signifier of many other factors from productivity to codebase health.

Attendees also discussed the challenges they have when it comes to metrics. A couple agreed that engaging engineers can be tricky; for metrics that require team input (for example quick pulse questions), it can be tough to drive high participation rates. And one attendee shared some skepticism around metrics, questioning their potential value.

How Code Climate uses metrics

Next, Brian Helmkamp, CEO of Code Climate, gave a presentation on how VPs can use metrics to communicate with their executive teams and boards. He shared why it’s important to use more metrics in engineering, and offered some advice for the types of metrics that might be useful to track. Here are his key takeaways.

• Many companies depend on their engineering orgs for future profitability, but engineering has lagged behind other departments like marketing, sales, and HR in the journey to becoming data-driven. Where sales leaders are expected to back up their conclusions withs numbers, tech leaders still often resort to ‘trust me’.
• It’s important that engineering leaders use data to talk about their work in a quantifiable way when communicating with non-technical peers, from CEOs to boards, to advocate for engineering and make sure it gets the seat at the table it deserves.
• All metrics are flawed; some are useful. With the right data, you can back up your conclusions and requests, cultivate a culture of transparency, solve problems quickly, and celebrate accomplishments and progress.
• If you’re feeling stuck, try thinking about these five types of data that might be helpful to track:
  1. Efficiency (how fast are we going?)
  2. Quality (are bugs being discovered?)
  3. Allocations (where does our time go?)
  4. Financial (where is the money going?)
  5. People (what’s going on with headcount and hiring?)

Group discussion

After Brian’s presentation, the group came together to discuss their own experiences of using metrics. First, they were asked to share examples of metrics that they currently use to report at a management board or executive level. While one attendee shared that they’ve had challenges using metrics to drive productive conversations, others said they’ve had success sharing data around infrastructure SLOs, performance, operational costs, committed vs delivered goals, team health, and people.

The group was also asked to share examples of the different ways they report their metrics. A common answer was regular meetings including bi-monthly catch-ups with executives, monthly all-hands, and bi-annual catch-ups with the board. One attendee said they share data with folks outside of engineering on-demand, while another creates weekly documents for the whole team and semi-annual decks for the board.

Finally, attendees were asked to share what they would like to know from other engineering leaders related to metrics. There were a few questions around getting metrics right: How do you identify targets for the metrics you measure? How do you create a cadence of reporting that minimizes babysitting? What can you measure that motivates engineers? There was a question around reporting up: How do you use these metrics to convince the exec team to invest more in technical projects? And there was a question around the usefulness of data: It feels like metrics are often used as air cover to defend pre-conceived ideas; how can we make these conversations more productive?

Reflections

As the conversation came to a close, the group agreed that using metrics is a challenge. It can be difficult to know which data is valuable to measure, and which metrics should be turned into goals. Sharing a final thought, Brian Helkamp reminded folks that while it’s always valuable to have an objective result related to a goal, it can also be useful to look at data even if it’s not target-related; try observing something for a quarter before assigning a goal, and it might help you to get unstuck.