Optimizing micro-feedback loops in engineering

Recently, in an article about maximising developer effectiveness, I analyzed the habits of low and highly effective engineering organizations.

One of the main driving concepts were feedback loops and being able to measure them. There are a number of key feedback loops that highly effective organizations have put a lot of effort into optimizing and maintaining. 

We analyzed what developers are actually doing all day, and what they are hoping to achieve with each task. It can be thought of as a series of feedback loops. Abstractly, developers make a change and then want to validate that they are on the correct path. They want confidence in knowing that they will hit the outcome they are looking for and are not wasting time. 

If they are not on the correct path, they need detailed information as to why not. This feedback can be from a colleague, a system, or a user. There are different size feedback loops: ones that happen multiple times a day and ones that are done every week or month. 

In the article, I list out a few typical ones. As a developer I want to:

• Validate that a local code change works
• Find a root cause for defect
• Validate that a component integrates with other components
• Validate that a change meets non-functional requirements
• Become productive on a new team
• Get answers to an internal technical query
• Launch a new service in production
• Validate that a change was useful to the customer

We found that there are stark differences between how low and highly effective organizations implement these feedback loops. 

Thinking about software development in this way allows you to think holistically about the value your tools, platform, and process are providing to developers. Too often I see engineering organizations jumping into buying sophisticated tools or creating very complicated solutions that take a long time to implement – usually, the shiny current technology of the moment. Somewhere along the way, the reason they are doing it and the value to developers get lost. Productivity can actually drop. Tools are important but bottlenecks will also reside in process, communications, and org structure. When we can analyze and measure the value chain in these feedback loops we can identify sources of friction. 

As an example, for a highly effective company launching a service, they will typically have a frictionless ‘paved road’ or ‘golden path’, where a new service can be stood up in production with observability and deployment pipelines in a few clicks. This is provided by a platform team that is owning this capability. Whereas in a less effective environment, it’s akin to a maze (without a map, and with people trying to send you the wrong way); teams are re-inventing solutions, taking manual steps, and going through endless amounts of bureaucracy to get something launched. 

Micro-feedback loops

The feedback loops could be very small – micro even, and they’re the types of validation that developers seek to achieve 10 or 100 times a day. They are very important to keep optimal as they allow the developer to stay focused and to be in that elusive state of flow. Too often, they are not appreciated enough, and instead focus on big programs or tools as the target of transformation. 

The micro-feedback loop I mention is a developer trying to do the first level of valuation for their code change. This would be in their development environment, meaning local on their computer or in a personal VM or cloud env. The validation the developer desires is that the code ‘works’ as expected. 

They validate this by compiling, deploying to a development application server, and then running in their client (e.g. browser or API client). They can also validate it by running a subset of the test pyramid that  the developer wrote and existing tests for that component. Every developer runs these loops many times a day and so it is important that we keep it fast. It might be small inefficiencies 10s of seconds each time, but they quickly add up. 

Fixing the small things has a massive amount of benefit and has compounding effects, including reducing the cognitive energy for developers – whereas previously they might have had many small breaks of flow through the day, e.g. interruptions and context switches. The improvements in the metrics associated with the micro-feedback loops will culminate in bigger ones, such as the four key DevOps metrics from Accelerate. 

Of course, this isn’t particularly novel; developers want fast work setups. The more unique part is having engineering management support and measure this, encourage optimization and provide time to do it, and identify technical KPIs that are lauded in the same way the product KPIs are. 

Micro-feedback loops can be applied to different scales, even smaller cycles. For example, when a developer is coding, what in-line help or what nudges are the integrated development environment (IDE) and the general environment providing? On this level it should be continuous feedback, almost instant, checks such as reformatting, spelling, lint fixing and running tests in the background. 

What do we mean by optimizing feedback loops?  The ones I mentioned are quite different, but the following are some common traits:

• Fast – If it is not quick, then the developer will get distracted and go onto another task
• Accurate – otherwise the developer loses trust
• Confidence building – the developer must see value in the loop; they should get confidence that the validation will prevent future problems from happening or improve the product 
• Easy to interpret – if the results are not easy to understand, the developer wastes a lot of cognitive energy interrupting it, or they have to do more work to uncover the real results

Information discoverability
A common source of developer friction is in discovering information. This can be in many forms: understanding an API you depend on, help with a particular architectural or coding pattern, or documentation on a library, for example. Trying to get useful information can be very frustrating for developers as often teams have not kept up-to-date information, have to request access or internal systems are slow and difficult to use. This means that developers have to resort to trial-and-error approaches. 

We are seeing companies investing heavily into the user experience of internal platforms. This helps to keep developers in flow. 

Spotify, as part of their investment into developer experience, have a team that actively works on discoverability, their mission has been to have ‘unstuck engineers’. They do this by intentionally designing their internal tools such as Backstage to make it easy for developers to get information at their fingertips. 

Gary Nieman, a product manager working with discoverability, gave two examples of this focus on reducing small friction points: ‘We recently re-imagined our Backstage Search – switching to a Mac OS Spotlight-like quick search to get engineers to what they need. The search success metric that we use is based on if the engineer comes back to the search page. So we want engineers to find what they want quickly and move on – and we measure that to further refine our search algorithms and UX.’ 

Another example he gave is the docs-as-code feature, ‘Backstage has the ability to highlight text on the page and directly add a GitHub issue. We then expose the doc site’s number of issues on the page. This is all part of the feedback loop that is so important to keep our documentation of a high quality and up-to-date.’

Gary says ‘I talk about engineers saying “I can’t find stuff”. Our job is to fix that. I jokingly say that I never want to hear that phrase mentioned and when it is mentioned – it is like a stab to the heart.’

Measuring productivity
A word of caution: often when we talk about productivity, it is focused on individual productivity – for example, using measurement in order to produce aggregated reports that managers can quickly scan. In the worst case, this individual productivity is focused on metrics that are too far away from any value to end users (e.g. lines of code, hours worked). 

While in this article, I am encouraging you to measure how effective the environment you are providing for developers and their day-today tasks in my mind, the team is still the most important construct. The most effective teams are ones that have the culture to encourage developers to help each other – mindful safe environments to fail and learn. Some output-based metrics will actively discourage this. This is important for making developers productive when they join a new organization or work in a new area. 

Summary
In this article we talked about how to think holistically about feedback loops and the value they are bringing to developers, to transcend tooling, process, and org structures. The importance is in reducing friction in micro-feedback loops, how those improvements will compound to product delivery and DevOps performance, and creating a culture that empowers that. 

While this article presents an introduction to feedback loops, this is a large topic that I will continue to explore further in my writing, with future articles delving deeper into practises and case studies of micro-feedback loops. 

Thanks to Gary Nieman, Pia Nilsson, Cassie Shum, Chris George, Roman Safronov,and Martin Fowler for their support in creating this article, and Kennedy Collins for the visualization.