Mastering large-scale migrations

Large technical migrations can be fraught with danger, here are some techniques to master these important projects.

Migrations can range from simple data transfers, to complex multi-year digital transformations. It can be as simple as one diff (remember those patch updates for NPM packages), to thousands of developers working for multiple repositories across time zones. As the scale of a migration increases, complexity increases exponentially, with large-scale migrations having multiple potential points of failure. 

That’s all a long way of saying that migrations are hard. They are challenging engineering problems that require grappling with complexity and constantly thinking about failure scenarios and how to rollback. And there’s no textbook to work from.

Why would anyone do a migration?

Considering all of those factors may have you asking: why would anyone want to do a migration project? Well there are several reasons why a migration would be desirable, or even unavoidable.

Migrations improve performance. For example, Java version 21 comes with a better garbage collection performance and a lower memory footprint compared to Java version 8. Moving to Java version 21 will not only enhance your developer experience but also improve your server costs. This type of migration will not only enhance developer productivity, but also make the organization run more efficiently and effectively.

Migrations reduce cost. Migrating from a single cloud vendor to a hybrid solution, where each service is optimized for the workload, or migrating a workload to an optimally priced service, can each involve a complex migration project. However, engineering leaders are being increasingly tasked with optimizing their infrastructure costs, especially during times of economic uncertainty and in the race towards greater profitability. Sometimes these projects take a long time and will increase costs during the migration. But, in the long run, they can really move the bottom line. 

Migrations improve your developer experience. Migrating to a different tool, language, or framework can increase the productivity of your most scarce resource: developer time. For example, if your front-end team is using a flow type system that is both error-prone and lower in performance, then migrating to a robust type system like TypeScript could improve your developer experience and reduce churn in the team.

Migrations improve scalability and security. Changing security and compliance requirements might lead to large-scale migrations. If not planned properly, these migrations can land with tight deadlines and require an all-hands on-deck approach. Remember the Log4J vulnerability?

What are the necessary steps to a successful migration?

While that may all sound pretty daunting, large-scale migrations don’t have to be painful if you follow these steps.

Plan 

Large-scale migrations might take just a few sprints, or years to complete. It is important to have a very clear vision before embarking on a migration journey. This includes, but is not limited to, a detailed plan of the milestones, timelines, resources, and cost. With proper planning, we can capture issues earlier and build contingency plans if needed. Anticipating failures and preparing for them is imperative at this stage. 

Document

Once you have a plan, you have to document and communicate it. The longer the journey, the louder communication should be. Document the entire process and get approval from your organization before embarking on the actual work. Documentation also encourages you to provide the right information and creates an opportunity to avoid duplicating work. Communicate the right amount of information with your stakeholders based on their criticality. This document is not a one-time event. It is a continuous process.

Automate and trial

Now that you are set with a documented plan, the next move is to trial the migration. Start with something minor and automate the migration procedure as much as you can. Automating as much as possible will increase the likelihood of success. Gather feedback and refine your automation script until you finish this trial.

Test

Tests are a vital element to the migration process. Every time you migrate something, it is essential to have tests to guarantee that what you are doing is accurate and confirms the completion. If you cannot validate via tests, then stop the migration and write tests. In no circumstances should you perform a migration if you do not have tests that validate the migration. 

Fallback/Rollback

Fallback/rollback should be an essential part of any migration. Being ready to reverse any changes if things don’t go as planned is important, as it’s not always possible to take the systems offline to carry out the migration. Make sure you have the control to make a swift, seamless fallback if necessary.

Lastly, don’t forget to complete the migration. Always remember, 99.9% is not enough with a migration, it has to be 100%. Incomplete migrations always leave room for further regressions and churn in the team.

What is your role as an engineering leader?

As an engineering leader, your job during this process is to ensure that:

• The migration plan is strong and contains all the steps necessary for completion.
• You have identified areas where the team can utilize automation even more.
• The team has all the necessary tests to determine the success of the migration. 
• Set a benchmark for the expected quality of the migration and address any risks.
• The team has a fallback strategy. 

You will need to ensure that the team is equipped to manage the entire migration process and can flex to changes in circumstances, such as the scope of the migration changing. This involves identifying when to onboard and offboard folks into the migration team. 

You will also be expected to be the source of communication with stakeholders. Frequently provide updates across the chain. Make sure that everyone is clear of why we are doing the migration and what is involved in it.

As a leader, nobody expects you to be a technical expert, but you can and should proactively identify and address issues when and where needed. Sometimes your senior engineers might be against the idea of an engineering manager reviewing the design document, but I would strongly suggest seeking and providing feedback whenever and wherever needed. 

When at an impasse about some decision, you need to be decisive on what path to take. 

But also note that life during the migration will be difficult and you should prepare your team for that. You should watch out for:

• Increase in development effort and timeline
• New features always being prioritized
• Legacy code and features proving difficult to migrate
• Difficulty testing the suites completely
• Applications become too slow
• Churn inside the team

Selling the project

Convincing your leadership on large-scale migrations can also be challenging. Focus on:

• Identifying the pain points. What are the specific problems that the migration would solve? For example, is the current system outdated or insecure? Is it difficult to maintain or scale? Is it costing the company money?
• Quantifying the benefits. Whenever possible, put a dollar value on the benefits of the migration. For example, how much money would the company save on IT costs? How much revenue would increase from improved productivity?
• Mitigating the risks. No migration is without risk, so it is important to identify and mitigate any potential risks upfront. For example, how will you ensure that the migration does not disrupt business operations? How will you protect your user data?
• Developing a migration plan. This should include a detailed timeline, budget, and resource plan. It is also important to have a rollback plan in case the migration does not go as planned.
• Getting buy-in from key stakeholders. This includes other departments that will be affected by the migration, as well as IT management. Having the support of these stakeholders will make it easier to get your manager’s approval.

Final thoughts

Remember: the pain can be worthwhile. A well-planned and executed migration can provide meaningful progress in an organization, reduce technical debt, and improve developer experience.