We are in the midst of a cross-organizational challenge where we’re trying to measure the productivity of our product and engineering teams. The endeavor includes around 120 teams with about 800 engineers. These teams have been split up into eight macro-divisions – or chapters. After much contemplation, we have created a Data Studio dashboard that incorporates DORA, cycle time, lead time, and throughput metrics to monitor company productivity. These metrics can be viewed at a team, solution, chapter, or company level.

Our CEO hasn’t received this all too positively, saying these metrics are "too detailed," making them unsure how to act on the information provided. 

It seems that we need to find some way to abstract these metrics into more actionable insights. Any advice for us?

Thanks, 

Asa

Hey Asa,

I have some general insights and some possible specifics that I want to dig into.

Align with your CEO on their expectations.

I’ll start with the specifics. The most promising clue for next steps is mentioned at the end of your message: the CEO’s perception of the data you’ve presented.

I want you to think of these reports as a product of sorts; their goal is not to be comprehensive in the abstract, but rather to solve the specific problems of their user, in a specific context (in this case, the user being the CEO or the executive team).

From that perspective, the first step is to understand what your CEO assumes so you can help validate those assumptions, identify any blind spots of theirs so you can paint a more complete picture, and how they define success so you can then find the right data around it.

How to learn more about your CEO’s vision

I would recommend starting a conversation with the CEO to better understand what they’re looking for. Here, share your observations on the current report where the data may not have explicitly answered their questions. From there, field your own broader inquiries to gain more specific insight, such as:  

• What do they mean by productivity? The CEO likely has their own image of productivity within product and engineering teams. This means that they’ll have specific results, activities, or numbers that they consider productive – and not. What are they? Uncovering them will give you a clearer picture of what specific data you can then research and present in the next version of the dashboard. It will also help build a shared language.
• Why does productivity matter to them? What gains are they hoping to make with improved productivity? Is it customer results? Growth? Engineering culture? With this context, you may be able to gain inspiration on the specific, additional data you can show in the next iteration of the report you share with them. The resulting product will be closer linked to the tangible improvement they want to see.
• What are their worries when it comes to company productivity? Your CEO will have some nagging concerns they hope to address with these metrics. What are they? Think of these concerns as assumptions or hunches the CEO might have. Once you validate them with actual evidence, you empower them with either something concrete they can take action on, or something they can rest assured is actually working well and does not need their attention.
• What’s going well? What areas within the broader organization do they think are currently productive or working “well”? Combined with your new understanding of the CEO’s definition of productivity, this can give you further clues about specific data or metrics to surface.
• How do they envision using these reports? Is there a workflow they have in mind? How would they use this material to make decisions? This context will help you think about structure and how to deliver insights.

Note the language in these questions; it’s interviewee-centric and gets at subjective answers. You’re trying to understand how they perceive the system, not how they should perceive the system.

The other goal this step will help you with is building trust with the executive team. By building a feedback loop of sorts, you establish a foundation where the executive team trusts you to understand what matters to the company because they see you invest time in it. This might seem a bit fluffy, but it can be the difference between a dynamic that resembles a partnership and one that feels like a disconnect.

Factors influencing productivity that are difficult to quantify

My more general observation on the metrics you mention above (DORA, cycle time, etc.) is that they focus on a narrow definition of productivity: output over time. These metrics do an excellent job at that, and, what’s more, you’ve made the dashboard accessible to various org levels to get a more actionable-by-a-specific-person view.

However, I wonder if that’s only one part of the picture. In an organization of that size, I would bet that there are other factors at play, that aren’t easily visible. To give an example, consider a company where deployment frequency is high, lead time and failure rate are low, but it takes forever to land on what features get prioritized. What’s more, customer feedback is not part of the process. Though the teams work efficiently, they could be doing so on something that’s not useful or even in need of a lot of rework. Ultimately, that’s not productive – and these are factors that can’t be easily quantified in a dashboard.

Here are some factors that could be affecting productivity:

• Time taken to reach a decision. Examples: How many people are involved in deciding an area’s roadmap? What about a feature? How long does it take for a strategy to pivot when new information arises?
• Speed and quality of feedback loops. Examples: How does feedback from sales or customer-facing support get to engineering? How often and with what delay is it acted on? How distributed is business insight in the company? Do you notice that business context is shared and up-to-date enough that teams are empowered to act independently and usually get things right, or is context mostly held by a select few who need to continuously be consulted – thereby becoming bottlenecks for others?
• Quality of cross-team communication and collaboration. In such a large organization, it’s very likely that multiple teams have to collaborate to ship a bigger solution. Examples: How frequently and how well does alignment across teams happen? How expensive is it – does it look like continuous surprise-and-realign (where a team goes down a certain path that’s misaligned with what another team is pursuing, only to find out months later that they’re pulling in different directions), or is it more automatic and continuous? Is there too much overlap, i.e., a high risk of duplicate work, or, conversely, too little overlap where there’s a high risk of shipping badly integrated features?

You’ll notice that a lot of these facets are hard to capture in numbers. To aid you here, consider visual communication tools. For example, if you have a complicated approval process in your org, paint a picture of what those endless back-and-forths might look like. It might tell a clearer story than just reporting that it takes seven days for teams to make a decision.

If you’d like some tool recommendations, consider Dan Roam’s work or a simple user journey map (with a twist: the customer is the CEO or other roles in the company, and the journey is the things they’re trying to get done).

Final thoughts

To be clear, the work and results that have gone into creating your existing dashboard shouldn’t be neglected. I’m not advocating throwing them away. They’re meaningful metrics for a partial definition of productivity. It’s just possible that it’s over-indexing on some factors and missing others that stakeholders – including your CEO – would also factor into the equation of strong productivity.

What you have now is version one of your productivity product. Combining your CEO’s expectations and a broader view of what it takes to build software at your company, you might find new ways to capture information that will lead to actionable outcomes.

–– Maria