Using observability to accelerate the Engineering Flywheel

Observability is not just a new approach to monitoring and alerting, it is a crucial component for application development and a north star for team leadership.

A lot has been said about observability, yet many companies still find it a challenging topic to address. For teams looking to implement observability for the first time, there are a number of considerations to be made; there are different ways the benefits can be measured, there are different tools and vendors, and there are organizational considerations as to who will own the final implementation. Complex issues like this can be made even more difficult when there are other efforts, from organizational changes to cloud migration, that compete for priority.

First, focus on what is important

As the CTO of a startup, competing priorities are very familiar to me. I speak with other engineering leaders and their experiences are similar. As we progress from being developers to being managers and leaders, we start to hear feedback from end-users first hand, we receive feature requests from Sales directly, and we have to think about how our development goals fit into the larger needs of the company. Implementing observability can seem like a procedural effort that needs to be juggled with these other demands. However, I see it as an effort that makes other engineering initiatives run more smoothly by helping engineers focus on what is important.

As engineering leaders, our biggest responsibility is to unblock the other members of our team.  While we are also responsible for making sure development converges on a useful application, the reality is that any vector of a solution that we initiate as technologists will be reimagined and reevaluated every day by the talented people on our teams. We will set a vision that guides release cadence and the quality of the product, but after that, our real job is to use our limited time to make sure engineers have what they need to solve the many challenges that line the way to that vision. Observability tools are a structural aid that help your team see and discuss the inner workings of your application, and allow new members of your team to join in that conversation much more easily.

Remember what you and your team are here to do

For people new to observability, the many discussions around the subject can make it sound like it is an end goal in and of itself. It certainly does not have to be and, unless you are on an observability team at a large social media enterprise, it probably should not be. These discussions are valuable though, and include some key points that other authors have already discussed in intelligent detail here on LeadDev:

• Observability is as much about increasing developer velocity as it is about reducing time to troubleshoot in production
• Metrics, logs, and traces are the pillars of observability and should be analyzed in a single, purpose-built tool that allows for ad hoc analysis (otherwise, it’s just monitoring)
• Observability should be owned by modern cloud dev teams since they are expected to own and run their code

An important consideration that has had less coverage is that your observability strategy should match your leadership strategy. If you want knowledge concentrated in subject matter experts, traditional and reliable methods can still work. If your product will differentiate on the infrastructure level and you want to build deep domain teams, including a robust DevOps team, you may want to own the expertise behind maintaining custom or open-source methods of observability. If you are managing one or more teams developing end-user features, you will want tools that can easily manage themselves to make observability accessible across the team.

The tools you choose for a team are a signal of what you find important. Some of these can seem obvious or commonplace, like code repositories or task management platforms, but they are still means of unblocking your team. Observability strategies are more unique in that they signal how you want to structure your team’s cognitive load (more on that here). For me, that means having a team that can focus on features and scales equally well. I still remember my first experience with microservices and how significant it was for me as an engineer to be able to build fast, get in front of customers quickly, and scale-out easily. Part of that was not getting stuck in the details of setup, so I value observability that is friendly to developer teams and is as low touch as possible.

Mastering a new environment

Microservices and the rise of container and serverless workloads is a major change for observability that deserves its own focus. This is a significant development that will continue as more teams look to accelerate their development cycles. After all, the best way to unblock teams is to allow them to work in parallel, and nothing lets them work in parallel better than microservices architecture. This new approach adds new challenges, however. As teams leave the monolith, they take on new complexity while leaving familiar development guideposts behind.

As teams make this significant transition, it is more important than ever to help them stay focused on delivering meaningful features. Observability can be particularly useful for reducing complexity by revealing the significant relationships in the microservices environment. As the number of containers, functions, and APIs grows, proper observability will help teams keep track of dependencies in their new architecture. Many teams will use third-party service APIs to quickly add queueing, payment processing, and other functionality to their applications. Proper observability will make it much easier for them to understand the complex integrations required to make some of these services work.

Achieving the Engineering Flywheel

When your team is unblocked by confusion and able to focus on creating new, meaningful features that scale with your customers, you can achieve the ‘Engineering Flywheel’. This is a team-wide flow state where engineers build faster and with greater reliability because they can see and understand their application. Observability makes it easier for individuals to achieve this level of understanding and to teach it to their colleagues.

When successful, your team will make more focused decisions and learn even more about what they are building in every phase of development.

• Design – your team will anticipate outcomes because of increased understanding of dependencies
• Implementation – even new engineers will be able to confidently commit changes if they can see what the impact will be ahead of time
• Test – developers will catch more issues before they ever reach customers
• Production – your team will rapidly resolve issues so your customers can get back to being happy and your team can focus on building the next great thing

When thinking about observability, you will need to consider different factors like tools, vendors, and team ownership. Since you will be choosing a tool that will guide the focus of your team, be sure to consider what you will want them to focus on and how you want to lead them.