Don’t cross the Rubicon: engineering practices you don’t want to delay

Back in 49 BC, Julius Caesar broke an ancient law forbidding any general to enter Rome with the army by crossing the river Rubicon.

By doing so, he declared war and did something that could not be undone. Ever since, ‘crossing the Rubicon’ has come to symbolize the point of no return.

As a tech leader at ThoughtWorks, a large part of my job involves recommending practices to our clients so they can build and deliver good quality software, faster. In doing so many times for many clients, I have created a toolkit that contains practical advice that may help you avoid the point of no return.

In production from day one

When I turn up on a project, whether it is greenfield or existing, the first thing I try to understand is the path to production. Just because going to production is the last task in the life cycle of building software, it doesn’t mean you procrastinate worrying about it. Flip it on its head and aim to achieve being in production from day one. By this, I don’t mean you need to be releasing from day one, but you should be deploying to production as soon as possible. Attempting to deploy to production as early as your first commit forces you to establish good engineering practices from the start.

This State of DevOps report identified four key metrics that have a direct link between organizational performance and software delivery.

To be in the high or elite bucket and successfully deploy on demand as described by the State of DevOps report, it takes a lot of investment at the start. Deploy a ‘hello world’ setup or your first feature into production as soon as possible. Progressing this to production helps you bring forward the risks and uncertainties associated with cross-functional requirements. It accelerates the:

• Design and build of infrastructure across all your environments which will help you achieve consistency.
• Design and build of automated continuous integration pipelines. This will require you to establish your quality gates within the pipelines if you know your commit is going up to production, even if it is toggled off in production.

An important distinction here is the act of deploying to production and not a release. A release is driven by the product vision and is a business decision to enable the users with new features. However, deploying to production frequently reduces risk of late integration, and the feature is disabled to the users.

Prioritize setting up a path to production that allows you to achieve higher deployment frequency over building multiple features that stay stagnant in lower environments. Apart from all the things you need to uncover and solve as a leader in tech, the first thing you want to kickstart is aiming to achieve higher release frequencies and reduce lead times. Ask yourself: what do I need to do to get to a stage where we can successfully deploy every commit to production? What battles do I need to fight now that will help us move into a high or elite performing bucket (see State of DevOps).

Each day that you are building something new, you are introducing new systems – so the longer you wait, the harder you are making it for your team to achieve higher deployment frequencies. Stop and reprioritize to achieve higher deployment frequency over building something shiny that doesn’t reach the users reliably. Don’t reach the point of no return where the effort involved to uncover and reach the elite status might be so hard that you choose not to do it.

Stop half baked CI/CD

Continuous integration (CI) and continuous delivery (CD) have been around for decades, and there are a lot of organizations that have a CI server in place which is a great start. I remember the first project I was on (back in 2012), and this was the release process the organization had at the time: we built an artifact on our local machines, zipped it up, uploaded it to Dropbox, then somebody picked it up and deployed it. And this was just the process for deploying to one environment. Never mind going to production.

Most organizations today have CI in place, but just having a CI server setup isn’t enough. Doing CD properly requires investment in good software practices.

CD doesn’t come for free with CI

There is a misconception around CI/CD. Just having a CI server in place doesn’t automatically grant you CD.

Code maintainability and architecture design heavily influence independent releases. They also play a part in how quickly you are able to introduce a fix or feature to get to market as soon as possible. It is practices like code maintainability, loosely coupled architecture and trunk-based development, that are much harder to retrofit once you have an established product. Cross-functional requirements (CFRs) like testing, monitoring logging, and deployment automation should be considered from the start too. They promote your organization’s ability to do continuous delivery well. They contribute to your ability to successfully and repeatedly push code to production with minimum manual testing effort.

If your current release requires a lot of coordination across multiple teams and systems, you probably want to identify what has become a blocker for you to continuously deliver your software.

Heavyweight process will hamper your ability to do continuous delivery

A lot of organizations have a lot of processes in place but they don’t invest in the right tools and technology. On the other hand, there are organizations that don’t have a lot of processes in place but don’t have the cash or time to invest in the right tooling. It is important for processes and technology to bounce off each other rather than fight against it.

Change management ‘predicts’ low organizational performance. Some processes exist because they always did. Nobody questions the status quo. And they existed because at some point in time it was the best solution in the absence of technology. But today, we have a plethora of tools and technology that can reduce our release process drastically. Heavyweight change processes are a hindrance to continuous deployments. Irrespective of what your organizational needs are, it is important to reduce the burden of manual change processes. Heavy change processes will bog teams down. It becomes easy to get into a place where you avoid releases because you have to go through a process. Consider embedding the process within CI, reducing manual overhead as a first step to help your organization gain confidence before changing it entirely.

You have incurred tech debt as soon as you write a line of code. It is going to cost you effort and time if you continue building lots of features with the hope of coming back to the good software practices like code maintainability, loosely coupled architecture, testing automation, and deployment – to name a few. Ignoring the software practices that feed into achieving continuous delivery will leave you at the point of no return. Invest time in setting them up from the start. Losing the ability for continuous delivery will have a huge impact on your four key metrics.

Quality is everyone’s problem

Gone are the days when developers wrote some code and chucked it over the fence for quality analysts (QAs) to worry about. This mindset has changed with modern software development. Quality is inherently everyone’s problem. I have seen excellent outputs from teams where the developers, tech lead, and QAs work together from ideation all the way to execution.

When I started on a previous client project, I found out their path to production was very short (deploy to non-prod and then subsequently deploy to production – of course, if the quality gates passed). And every commit was pushed to production. Lead time for change was 20 mins with some exhaustive quality gates (functional and cross-functional checks) on the way. This really changed the entire team’s mindset.

• We never carelessly pushed and hoped for the best. Broken builds were addressed straight away.
• Because we relied so heavily on tests of all levels across multiple components, our code quality was very good. We didn’t write code that couldn’t be tested. This made our code easily extensible too.
• Because you are making small commits, it is very easy to identify what could have caused the problem when something went wrong. This meant the ability to recover from the problem was much higher.

In a nutshell, we, as a team, owned the quality of our code and understood it stemmed beyond handing it over to QA in silo and moving on.

Developers and QA pairing

Include QAs in ceremonies prior to development, like story analysis, iteration planning, and backlog refinement. And once the story is in play, developers and QAs should pair often to achieve the same goal: a good quality product. Because by the time its development is complete, it has had multiple eyes with different hats on the story, which massively reduces manual testing post-development.

Encourage QAs to pair with developers. This allows QAs to understand lower-level tests like unit tests and integration tests that are integral during the software development stage, thereby, again, reducing the duplicated effort of manual testing later. The test cases are also benefited by the keen eye of a QA who is able to input at an early stage. 

Encourage QAs to build automation capabilities where they can rely on building some layers of test independently to remove the grind of manual testing. A good example of this is the functional testing layer. I have seen many QAs feel comfortable and contribute to this layer of testing.

Not just quality of code

‘Quality’ doesn’t just mean the quality of the software you write. It’s equally important to build a quality product that your users would find beneficial. For a previous client, we regularly used hypothesis-driven development to build features for the users.

This is an area that requires team discipline and relationship building which takes a long time to establish. Along with good software practices, focus on establishing good team culture. Don’t reach the point of no return where habits become difficult to change.

Don’t take your business hat off

As a tech lead, you are not only responsible for the technical vision but you are also responsible for understanding the product goals and supporting the product vision via your expertise – which happens to be building and designing software. It is important you always have your business hat on because you are best placed to advise on the feasibility of a feature. Product vision goes hand in hand with the technology supporting it and it is your responsibility as tech lead to inculcate this mindset within your team.

Build a culture of ‘Why?’

Encourage your team to question why they are building what they are building. It is important that they feel comfortable questioning the business requirements and questioning technical design and solution. It particularly helps when you, as a tech lead, lead by example and always bring the business requirements to the forefront when making decisions.

You will be surprised by the impact your team can have on the product when you give them the space to question and provide solutions.

Not all solutions are tech solutions

We developers inherently tend to be nerds. We love a challenge, so it’s not uncommon for us to jump into solutioning as soon as we encounter a problem. It’s exciting for us to find a solution to something that hasn’t been solved before. However, tread with caution. Take a step back and ask yourself if the solution has to be technical.

On one of our previous client projects, our developer team identified an edge case which meant the client was losing some money. The team started problem-solving and spent hours going through pros and cons, and liaising with the multiple teams they integrated with. When this was taken to the business stakeholders, they said it was such an edge case that they were happy to take a hit instead of investing time in fixing this right now over other competing priorities.

Circle of trust

All my successful projects have been when this circle has worked seamlessly, with each person playing off each other’s strengths and skills. In today’s world, technology is at the core of the business and it is important for the developers to understand the business requirements.

Your team may not have all the roles in the image, but the important thing is that you are constantly working closely with the roles that are relevant to your product. And you may be liaising more with some roles than others. You should be joined at the hip and combining your efforts to accelerate the product build.

Invest time in establishing some of these working relationships as soon as possible. Don’t neglect the importance of staying close to the business and keeping your business hat on at all times. It takes time for a team to change working models; don’t reach the point of no return where a team is happy to work in silos or in partial silos.

Conclusion

Unfortunately, not every project has the luxury of starting fresh, so don’t miss out on the opportunity to establish good practices at any point in your product lifecycle. But do bear in mind the cost of delaying is much higher. ‘Don’t cross the Rubicon’ if you have the opportunity to start on something that is in its early stages. If you are working on an established project, product, and/or team, I highly recommend you evaluate your software delivery performance as you might be able to recover by paying some of the price for it now before reaching the point of no return.