Estimated reading time: 6 minutes

When an internal tool becomes indispensable, the real challenge is knowing when to keep it in-house and when to unleash it on the world.

When Spotify open-sourced its internal developer portal Backstage in 2020, even its most passionate advocates might not have expected it to take off like it did. The project has gone on to spawn hundreds of plugins and has been adopted by companies like Siemens, Expedia Group, and Zalando.

With 100% internal adoption at Spotify, why is Backstage so adored? In building a standard inventory to catalog internal development tools, frameworks, libraries, workflows, and APIs – paired with an open ecosystem – Spotify’s engineers landed on a genuine, underserved developer need.

When homegrown tools solve widespread problems, externalizing them has several potential benefits, from open source cachet, to commercial gains.

Spotify is not alone in externalizing successful internal platforms. Consider Lyft’s Envoy, now one of the most widely adopted service proxies. Or, Google’s Borg, which became Kubernetes. The list goes on – from Meta’s React and PyTorch, to Netflix’s Spinnaker, LinkedIn’s Apache Kafka, and so on. More recently, Block open sourced Goose, an AI agent framework for connecting large language models (LLMs) to operational systems.

But what defines an industry-altering internal platform? And as an engineering leader, how do you uncover these diamonds in the rough and champion their adoption?

It must solve a business problem

First is realizing what to externalize. According to Pia Nilsson, senior director of engineering at Spotify, the ideal candidates for externalization are “battle-proven internally” and move an important business metric.

For instance, to compete with the likes of Apple or YouTube, Spotify’s core objective behind Backstage was to enhance developer productivity and reduce context switching. “Developer experience was absolutely critical for Spotify,” says Nilsson. “It’s the only way we can win – every developer needs to be incredibly effective. We knew that YouTube had 20 engineers for every one of ours, so we had to optimize efficiency.”

The best tools don’t introduce additional friction – they streamline existing workflows. “What we’re trying to do is find the common developer journeys and optimize and automate them,” says Nilsson.

It should increase harmony

Spotify has a very autonomous engineering culture. Developers are given a lot of freedom over tools, which can lead to fragmentation without shared standards. Nilsson admits when she joined Spotify after leaving Klarna in 2016, the pendulum had swung a bit too far toward autonomy.

Striking a balance between autonomy and alignment is key. Before introducing Backstage, formal job roles and leadership structures weren’t clearly defined or consistently followed, resulting in many informal leaders. ”It was very difficult to get any kind of decision made because anyone on the team could just say, we’re not doing that. The Swedish consensus culture had gone nuts.” Given Sweden’s strong emphasis on egalitarianism and collective decision-making, these values can carry over into workplace culture.

It wasn’t uncommon to see different tools or passion projects in use throughout the company for a similar software development action, like authentication. Backstage helped Spotify balance its creative culture with a bit of accountability and order, reducing this type of duplication.

It should be relevant to most

The game-changing platforms are applicable across engineering disciplines, says Nilsson. 

At Spotify, they seek to optimize a common developer journey, like finding an API. If you can optimize a universal process like that, you exponentially reduce friction for all.

Of course, a degree of prioritization is usually necessary. “You start where it really hurts,” says Nilsson. For instance, Spotify’s internal platform engineering efforts first focused on tedious back-end tasks and then worked up the stack to support front-end engineers.

One of the challenges of open-sourcing an internal tool is making these company-specific elements universally adoptable. “There are absolutely common challenges in platform engineering, but every company needs to be met on a much more personal level,” says Nilsson.

With Backstage, this extensibility takes the form of community-driven plugins, which help extend the platform into niche areas. The Backstage plugin ecosystem now hosts 250 open-source best-of-breed plugins for things like continuous integration and deployment, container integration, observability, and plenty of others.

Successes should be measurable

You also need evidence the solution in question benefits most users. At Spotify, the more frequent Backstage users are 2.3 times more active on GitHub – an obvious sign of its utility. Of course, correlating the number of commits to productivity gains is complicated and could include false positives. But Nilsson believes it’s still a worthwhile indicator. “Where there’s smoke, there’s a fire.”

Although Spotify follows DORA metrics such as deployment frequency and ‘lead time for change,’ tracking these alone is often insufficient. They should be paired with more context-specific metrics connected to the change the organization wants to drive, says Nilsson. “For example, when rolling out our AI support bot and code assistance across IDEs, we’re measuring acceptance rates by humans and the bots’ confidence rate in answering.”

Another example is migrating to a cloud-based monorepo. “We’re able to pre-measure the whole migration since we have stored all git operations and can replay them in a staging monorepo,” says Nilsson. “This type of change-aware devex metric helps us predict how a change will impact cycle time, and we’re able to verify our assumptions, too.”

It needs support

Then there’s the question of timing. How do you tell when a passion project becomes a feature worth committing resources to? At Spotify, it’s about determining the longevity of the project and its reliability guarantees. 

For instance, a passion project that enters production and outlives six months can no longer be treated as experimental, says Nilsson. It needs strong reliability and to meet service-level agreements. In short, internal platforms require the appropriate support if they are to become company-wide heroes.

Keeping the success story alive

Many industry standards and open technologies that we now take for granted were initially developed to solve common internal development issues. By open-sourcing them and supporting their growth, each company positioned itself as an industry leader.

Organizations that externalize internal engineering projects often capitalize on their intimate knowledge of the technology by offering managed packages or enterprise support on top of the open-source versions. In Spotify’s case, they’re commercializing Spotify Portal for Backstage, which layers developer experience enhancements on top of Backstage and simplifies deployment.

That said, these projects require continual focused alignment. For instance, although Spotify’s internal platform engineering efforts have grown substantially, balancing standards and developer freedom remains challenging, reiterates Nilsson. “It’s a constant balance.”

Another reality is it can be a struggle to keep in step with your open-source success story. Although Spotify’s aforementioned software-as-a-service Portal for Backstage runs the open-source Backstage at its core, the internal version of Backstage is distinct and has evolved over the years. According to Nilsson, migrating from their internal version is challenging, as it supports many critical paths internally. It can also take effort to support new plugins as they emerge.

“Soon, it won’t be any different from the open-source version,” says Nilsson. Keeping parity with the open-source version will help avoid duplicate maintenance work, allow Spotify to easily adopt community innovations around new plugins, and better align the company’s commercial offering. “It’s heart surgery,” she says, but necessary to keep Backstage aligned with Spotify’s needs and the broader developer ecosystem.

Get articles like this in your inbox

Choose your LeadDev newsletters to subscribe to.