You have 1 article left to read this month before you need to register a free LeadDev.com account.
Estimated reading time: 6 minutes
Most engineering leaders are familiar with developer productivity frameworks, which seem to emerge with increasing frequency.
DORA debuted in 2018, followed by SPACE in 2021, and DevEx in 2023. McKinsey also took a stab at it… to mixed results.
“The big question is, what should we actually be measuring?” asks Abi Noda, cofounder and CEO of DX, a developer intelligence platform. Their answer to the question is DX Core 4 – a framework that aims to bring together key elements from those three frameworks into a single cohesive model.
Competing developer productivity frameworks cause confusion, especially since each caters to different goals. DORA offers prescriptive metrics focused on a software team’s delivery and performance, while SPACE supports custom metric creation, but lacks clear implementation guidance. DevEx is more focused on self-reported developer experiences.
To read more about the rise of DORA and SPACE, check out LeadDev’s Engineering Team Performance report 2024
Rather than replacing previous models, DX Core 4 wants to bring together engineering metrics, subjective viewpoints, and financial outcomes. This way, the framework can help inform various roles across the organization – from engineering managers to CEOs and CFOs.
Exploring DX Core 4
DX Core 4 was spearheaded by Noda and Laura Tacho, CTO of DX, in collaboration with the authors of DORA, SPACE, and DevEx, including Dr. Nicole Forsgren, Dr. Margaret-Anne Storey, Dr. Thomas Zimmerman, and others.
To provide a balanced picture of productivity, the designers arranged the DX Core framework into four broad dimensions: Speed, Effectiveness, Quality, and Impact. Each of these categories has one key metric and three secondary metrics.
Interestingly, DX Core 4 includes several self-reported metrics, like perceived rate of delivery and perceived software quality. Noda compares this to how a physical endurance athlete measures their perceived rate of exertion. For example, if a runner has a poor night’s sleep, their heart rate is higher during workouts the following day. Similarly, perceived metrics offer more context to better incorporate the human side of the equation.
Speed
In the Speed category, the key metric is diffs per engineer, defined as a measure of pull or merge requests. It’s important to note these are not tracked at the individual level. Of the three secondary metrics, two are borrowed from DORA (lead time and deployment frequency), while perceived rate of delivery is unique.
Effectiveness
Under Effectiveness, the key metric is Developer Experience Index (DXI), which is calculated using a research-backed 14-question survey to measure engineering performance drivers, similar to Gallup’s Q12 Employee Engagement Survey. An important caveat is that the exact questions used to compute DXI are proprietary to DX.
A unique secondary metric under Effectiveness is regrettable attrition. This measures developer churn, which can indicate the overall health and satisfaction of a team. Another interesting metric is time to tenth PR, which attempts to normalize the onboarding journey a bit more than looking at someone’s ‘first commit,’ since this is often artificially quick due to housekeeping actions like first sign-up or account registrations.
Quality
The Quality section includes another standard DORA key metric: change failure rate. Secondary metrics include failed deployment recovery time, similar to DORA’s ‘time to restore service.’ Other secondary metrics resemble elements from SPACE, such as perceived software quality and operations health and security, to measure satisfaction and well-being, performance, activity, and collaboration.
Impact
Compared with other productivity frameworks, Impact is the most unique dimension of DX Core 4, as it frames software development in direct business terms. The key metric is the percentage of time spent on new capabilities, which indicates if developers are innovating or addressing technical debt. A secondary metric is initiative progress and ROI, which is a measure of the costs and return on investment for individual projects. This would gauge the success of specific engineering projects and inform spending allocations.
The other secondary metrics are directly correlated to financial outcomes: revenue per engineer and research and development as a percentage of revenue. These are intended to be averaged and measured at the organizational level. The figures will vary depending on company revenue and the sector in which it operates.
Seeing the benefits
To date, tech companies Dropbox, Block, and Vercel have tested the DX Core 4 framework. The biggest benefit they’ve reported has been better organizational alignment and clarity.
Following these metrics can inform investments into internal tooling, says Noda, which can reduce friction and wasted effort. Vercel, for instance, has reduced cycle times by close to 50% in the past 12 months by watching these signals.
DX has also found an impressive correlation between a high DXI score and business outcomes. “For every one-point increase in DXI score, you save about ten minutes per week per engineer,” says Noda. Most developer teams want to reduce technical debt and improve code quality, but often struggle to translate that into business terms. The hope with DX Core 4 is that by tying engineering friction points to dollars, engineering can advocate and prioritize investing in these improvements.
Avoiding negative reactions
Of course, there’s the elephant in the room: developers don’t like being measured. “Engineers don’t want to be reduced to lines of code or number of pull requests,” says Noda. “From an organizational standpoint, balancing the human side of the story is really important.”
Consultancy McKinsey’s 2023 article, Yes, you can measure software developer productivity, was met with heated scrutiny from programmers since it prioritized effort and output over the sociotechnical aspects of software development. Productivity frameworks have received criticism for being reductive and easily gamed. Is DX Core 4 any different?
The key is not to measure at an individual level. Doing so can cause inauthentic results since some might inflate metrics, like maximizing individual pull requests, to skew results for personal gain.
“As soon as it becomes a target, it ceases to become a good measure,” says Noda, echoing Goodhart’s Law. Workplace productivity gamification at the individual level has also been found to cause privacy concerns and evoke an unhealthy sense of competition between employees.
Instead, engineering leaders should measure these cues at an organizational level and reiterate to individual contributors that the end goal is to improve their workflows. “Show up as an ally to developers, and make sure these metrics are communicated that way,” says Noda. It’s a good practice to counter-balance productivity signals with other insights, like developer experience metrics.
DX Core 4: A unified view
Developer productivity frameworks are imperfect, but leadership will always require some metrics to gauge the performance of the business. If executed properly, DX Core 4 is a promising new option that the entire organization can get behind.
