Preventing process deadlock

The worst days as an engineer are the days when you realize there’s nothing you can do to move your project along.

Your teammate is too deep into their work to review your pull request until later. A product manager is asking to try out the new feature you just finished, but there are stability issues with your testing environment. The feature that you built last quarter is still waiting for a legal review. Later that day, your other teammate asks for help debugging an issue, but the test environment seems to be working again and you need to get your feature setup before something else goes wrong. Everyone on your team is experiencing the same frustration you are; there seems to be an endless cascade of blockers slowing everyone down.

From a leader’s perspective, the team’s velocity is dropping and morale is falling. Your development process is headed towards the real-world equivalent of a deadlock. Your first instinct might be to gather everyone in a room and identify what everyone needs to get them back on track. This isn’t the wrong instinct, but this is the teamwork equivalent of restarting your computer when something goes wrong. Ideally, you wouldn’t want to restart your entire computer, much like we want our team to return to running seamlessly without constant intervention.

How do we go about fixing these blockers? Let’s take some cues from coding

People and processes can’t be debugged like code, but sometimes it’s helpful to take cues from programming solutions when we’re evaluating real-world problems. We can assume most software we’re writing involves more than printing ‘Hello, World’. We’re dealing with complicated stacks that involve reading from multiple databases, writing to queues, running complicated business logic, and making requests to different microservices. This is all being done with the expectation that we can maintain five nines of uptime, and with strict latency requirements.

In addition to the complex architecture of the modern technology stack, we also have product requirements, code reviews, testing plans, designs, legal and compliance sign-offs, security audits with corresponding product managers, designers, engineers, QA engineers, lawyers, and analysts who all need to collaborate. Both software and software organizations deal with concurrency, whether that is juggling different tasks in a program, or juggling different requirements in real life.

One way you might speed up your code is to look for areas where execution is blocked – usually because a resource your program needs is unavailable. For example, Thread A might be holding onto a database connection for too long unnecessarily, when other threads need to be able to read from the database to continue their own execution. A possible solution would be for Thread A to let go of its database connection while it’s doing other work, and ask for the connection again later. Thread A will be slightly slower at executing its own work, but it will be unblocking the rest of the threads that need the same connection. 

Let’s take this unblocking method of problem-solving and reapply it to a product development team. For this team and many like them, frequently delayed code reviews are a problem. While they are critical for maintaining software quality and helping engineers improve their own skills, they can also become a roadblock to a team’s velocity. A completed code review is necessary for the project to move forward, but is also a non-glamorous task that is often put on the backburner for something more immediately important. If one engineer is the most knowledgeable about one part of the codebase, the rest of the team is relying on them to review the rest of the team’s code, even though this engineer has their own work to complete. Dropping everything to review their team members’ code now would only delay their responsibilities to their primary project. However, delaying these code reviews has now stopped other engineers from moving forward. We also have to consider that down the line there are QA engineers, product managers, and actual customers that are impacted by this delay. If the engineer had instead prioritized code reviews, several engineers would be unblocked while only pushing back their own work for a small period of time.

What are some other frequent blockers?

Lack of organization

The product requirements and designs are spread out in a myriad of places and can be difficult to locate when you need them. A ticket for a feature references ‘designs’, but doesn’t actually attach them. Now our engineer is spending time chasing down documents and waiting for an answer rather than executing. Tickets should be organized and proofread by the person who created them prior to being sent to the engineers. This will save hours of time and allow fixes to be created more quickly. 

Outdated, or lack of, documentation

Properly documenting our software and workstreams is a necessary task that is often neglected. It can easily be thrown on the back burner or forgotten about when we are in a time crunch. If the documentation for setting up a development environment is out of date or incorrect, then future engineers may run into a multi-week affair of chasing down updates. Their quick onboarding could become weeks-long while they search for the right information. Moreover, the team often loses institutional knowledge that they have to work hard to regain again.

Develop an unblocking culture

Once a team has identified their blockers and adjusted their process, it doesn’t mean they won’t slow down when a new persistent blocker shows up. We need this thinking about unblocking to be a consistent frame of mind so that we can identify new blockers quickly and address them. To do this we need to emphasize team deliverables over individual deliverables. Get your team used to asking: how am I blocking someone else from achieving their goals? 

Here’s an extreme example – maybe there’s only one person on the team that knows how to deploy your service. They should be thinking to themselves that they should prioritize deploying code over their normal work, since the entire team is dependent on them to successfully deliver features. They should also prioritize figuring out how to get multiple people involved in this code deployment, so that the entire team cannot be blocked by a single engineer.

While engineering leaders should help identify these blockers, the onus should be on individual engineers (or team members in general) to dig into the less obvious cases. Building a culture around unblocking and empathy helps everyone see the struggles that others are having. Encourage your engineers to think about what tasks others rely upon them for. Praise them for not only completing their own deliverables, but also for unlocking the potential in others.

This push for unblocking as a culture becomes more important as teams become more distributed over time (a process that has only been accelerated over the past few months due to the pandemic). When the team is having fewer face-to-face interactions, some patterns are more difficult to recognize. Individual requests can easily be forgotten about when the engineer isn’t sitting next to you. Everyone has a responsibility to highlight patterns they see that could be slowing the group down.

Are there other applications to unblocking?

This concept of unblocking can also be applied to career growth and mentorship as well. What can I do to unblock someone’s career growth with regards to their communication skills, their technical abilities, or their visibility in the organization? Is there something I’m actively doing to prevent someone from growing?

Imagine there’s a senior engineer on your team. They can cover a large part of the stack and can complete things in a short amount of time, but they’re also prone to pulling “heroics” i.e.  taking on large, complicated tasks by themselves. This seems like a good thing for the team, as productivity is really high due to this star engineer’s execution. However, their heroics are also blocking the more junior engineers from learning new skills and taking on a larger scope. Additionally, the senior engineer is frustrated because they are pulling most of the weight on the team and not getting to develop their own skills. The team should help that engineer unblock the growth of the team by helping them realize that their heroics aren’t benefiting anyone in the long run. 

Thinking about your team through the unblocking lens may help you develop solutions that improve the quality of your product, your people, and your work experience.