Cheerful deployments with CI/CD pipelines

Continuous integration and deployment has come a long way in the last few years.

It was not long ago that CI/CD was an alien concept for most organizations. Managers and leaders talked about faster, more reliable deployments and releases but what it all meant in real terms, or how to get started, was not comprehensible to a lot of technical decision-makers.

What we now take for granted because of the prevalence of CI/CD in modern tech companies would have been considered ludicrous and even impossible only a decade ago. The point being, CI/CD philosophy and tech has advanced substantially within a short period of time – but there is still a long way to go.

Anyone who understands the importance of ‘deploy fast and robust or die’ understands the significance of functional CI/CD pipelines in an organization. Mature and reliable continuous deployment pipelines are a standard that every organization must work towards, but it is also important to remember that reaching a certain level of maturity and reliability requires commitment, investment, and patience.

If done correctly, CI/CD pipelines have a tendency to become the backbone of an organization. They not only define the technical excellence for an organization, but also lay out a cultural road map that leads to maturity. If not done correctly, these pipelines can paralyze an organization and cause more harm than good.

In the paragraphs below, I will list some cultural and architectural aspects of the CI/CD pipelines that are often overlooked but are vital for cheerful deployments. This is in the sense that they uplift the spirits of an organization and the confidence of the development team by doing what they are meant to do – i.e. to take code to production in the most reliable fashion and in the minimum amount of time.

Visibility in pipelines

Adding visibility or transparency to the deployment pipelines is one of the key attributes that any organization must aspire to achieve. Visibility brings cheerfulness to the whole process.

When a pipeline gets executed and deploys code to the production environment but fails to provide visibility and transparency to the development team, the following questions will be raised:

• Who initiated the deployment pipeline?
• What triggered the execution of the pipeline?
• When did the pipeline get executed?
• How long did it take to execute?

These questions imply that you are looking at pipelines that are invisible and increasing the opacity and confusion in an organization in comparison to having no pipelines at all. There are numerous aspects of a deployment that must be monitored and visualized, and in order to create visible and transparent pipelines, they must be integrated with:

• Meaningful dashboards
• Timely notifications

These dashboards and notifications give visualization to the pipelines and provide all the basic stats that are associated with the deployment. They can also assist in predicting the health of future deployments along with identifying the areas of application that may be weaker.

Monitoring and visualization is a way for any CI/CD process to communicate with humans. Things that are visible are easy to comprehend.

I have a saying: ‘Don’t fly blind in a world of frequent code pushes’.

Security in pipelines

‘There is no cheerfulness in not being secure.’

When someone talks about the security of deployment pipelines, I imagine a situation as sensitive as a cauldron of hot, molten lava bubbling inches from a collection of Van Gogh’s best works! An organization can put all the effort into making the production environment as secure as humanly possible, but if the deployment pipelines are not secured, it jeopardizes the health of the entire system. Your best work could go up in flames!

Deployment pipelines have access to all the internal development systems, from the codebase to the cloud environment, and from the artifactories and registries to the internal communication channels and email addresses.

If these pipelines are compromised, they can provide any external attacker a clear path into development, test, and production environments. The external entity could steal data or intellectual property, inject malware anywhere into the environment, DoS the systems, or cripple an organization’s ability to respond to an attack by freezing the pipeline itself. 

Deployment pipelines effectively extend the attack surface of the production system to the build and deployment environment. It is important to perform a threat model on the deployment pipelines along with other environments.

Always remember: ‘Security isn’t a problem until it’s a problem.’

Trustable pipelines

The deployment pipelines in an organization must be reliable enough to gain the trust of not only the development team, but also the product people and the leadership. These pipelines must be set up with the intention that everyone in the organization should be able to trust a process to get an update on the health of the deployed product.

Unpredictable crashes, errors, and long execution times are a few of the causes that take the trust away from these pipelines and replace it with contempt and disdain. 

It also must be remembered that it takes time and patience to develop this trust. If for some reason this trust is not taking shape or is shaken due to some serious incident, it results in not only technical changes but cultural changes that are not beneficial for the organization in the long run.

If the development team does not trust the deployment pipelines to deploy the recently made changes, the developers will start performing deployment locally from their machines. Now, this is a dangerous situation, and will most probably result in the deployment of the snapshot of code that resides locally on the machine of a certain developer (because maybe they forgot to push the code to a remote repository). 

If deploying locally is easier, faster, and more reliable in comparison to using the pipelines, people will choose the convenient option. When deployment from local machines becomes a norm – i.e. enough people are avoiding the pipelines and choosing to deploy manually – then this is a cultural catastrophe with serious consequences. 

Time, resources, and effort must be invested in establishing trust in the deployment pipelines so that the pipelines are not only trusted, but also loved by everyone in the organization. 

Nimble pipelines 

CI/CD pipelines must be sprightly and energetic as opposed to being bloated and lazy. ‘Nimble’ is a hard word to justify for pipelines, but these are the pipelines that either maintain the duration of their execution or become perceptibly faster with time. 

If no new functionality is added in the pipeline but the execution time for it increases significantly, then this is a stink that must be taken care of as it indicates that the pipelines are getting bloated and need to be optimized. It is critical to maintain the analytics on the execution time of the deployment pipelines. There can be multiple factors behind the increase in the execution time, including for example: 

• The application that is built in the pipeline gets bloated – for example, dependencies of the application increase in number and it takes longer to download them; 
• The pipeline itself gets bloated – for example, if docker images are created in the pipeline and caution is not taken in creating optimized docker images, they can get bigger with time and will eventually affect the execution time of the deployment pipelines. 

If the pipelines are not bloated and are able to maintain their zestful nature, they are an affirmation of good code quality and coding standards. 

Conclusion 

These are only some of the architectural principles of reliable and mature deployment pipelines that should be considered; there are many more and they are highly dependent on the product and organizational culture, and they must be considered accordingly. 

There are challenges ahead in this industry which is going through a phase shift in terms of faster release cycles and better quality in the product. As long as people and organizations driving this shift continue to challenge the status quo and not forget the tedious months-long release cycles that started this phase shift in the first place, I have no doubt in my mind that mature and reliable CI/CD deployment pipelines will be a standard in the future.