We halved our continuous integration pipeline

Estimated reading time: 8 minutes

Key takeaways:

• A 64% CI speedup didn’t hold. Growth in tests, merges, and AI-assisted development outpaced the fix within months.
• Offloading the right tests beats chasing raw speed. Splitting critical from deprecated end-to-end tests cut runtime and flipped the pass/fail ratio almost overnight.
• CI speed needs an owner and a target, not a one-time fix. Treating it as a recurring product metric is what kept the gains from disappearing again.

---

It was in May when one of our staff engineers tossed out a number in our Slack channel that put an end to an internal debate I’d been having with myself. Roughly only 25% of the runs on our merge train were completing successfully. A majority were failing due to flaky tests or got canceled since another run that was ahead of them failed.

The queue had 50+ runs at its peak level, and some other developer was in it for close to two hours and still was 14 behind in the queue. In our software company, with a total of 400 people, more than 200 of whom are developers, shipping a change had become a gamble.

The problem here is that the same pipeline was actually our success story for the year in 2025, where we’d brought down the time of the pipeline from an hour to 22 minutes. Here is what happened after that, and how I realize now that continuous integration (CI) speed and stability is the bill you keep paying.

The decisions we made that led to the first 64% cut

Arithmetic became the cornerstone of the 2025 program since it allowed getting the budget approved. Working at 45 merges and 340 pipeline runs per day, every minute of merge-train time represented roughly six engineering hours of blocked time daily. Arithmetic helped us to turn “CI is slow” into a purchase order and made clear our first step – biggest saving first and cheap fixes concurrently.

Our first savings would be achieved by using the right infrastructure and, we consciously decided to go for the slower solution. We calculated runtimes of ten to 20 runs on three different Amazon Web Services (AWS) instances: C5, C7a, and C7i.

C7a, an AMD-based instance family, finished about 90 seconds faster on average than C7i, but its runtimes varied across a ten-minute range, compared with six minutes for Intel. As a result, we selected the more consistent Intel family of instances and increased the number of workers, which saved us approximately 12 minutes per run for 2.5% more costs.

No single change made the difference between an hour and 22 minutes, however. We split the pipeline into two phases such that linting, unit testing, and security scans, which yield results quickly because most failures happen in the first eight minutes, did not make authors wait for the full build cycle.

We stopped putting a browser on all test machines if it was only needed by some suites of tests, splitting them into browserless workers and saving around 0.5 GB of RAM per worker. A coverage scan would break the build if the split reduced the coverage in any way.

Retrying became a practice that nobody even doubted; as such, we automated the disbelief: any test that had three unrelated merge requests fail within half a day would enter a 24-hour quarantine period, and a large language model (LLM) costing $100 per month would analyze the error logs.

It compares each failure log with the corresponding code change and flags potential flakes after the first failure instead of waiting for the third. The success rate of the merge train improved above 90% from the lowest point of 82%, while 150 wasted retries per month evaporated. The tried-and-true quick wins were not absent either – cache linters, refactor a call to a deprecated metric that cost 120 seconds down to 20 – a total of three and a half minutes saved for three and a half days of work.

The bill for all of that increased by $1,680 a month for infrastructure, which is ~10%. The payoff: over 1,300 engineering hours a month, with only 30-50% of blocked hours considered lost, valued at over $65,000 even considering a low hourly rate of $50 an hour. It went from a one-hour process to a 22-minute one, which is a 64% reduction. I declared victory and moved on. That was the mistake.

Why our pipeline slowed down again

The pipeline was not broken, and we kept succeeding through it. Over the next six months, the test suite on the backend grew from 2,338 to 4,100 tests, while the test suite on the frontend tripled in size, and merges per month doubled.

Then came AI-assisted development, when growth became decoupled from headcount: the pipeline volume on our main repository increased over fourfold at peak, compute capacity grew 7%.

All of the mechanisms introduced in the 2025 program were operating, the stable instances, the split workers, the quarantine bot, but the workload had merely doubled around them. The “build and test” question in our quarterly developer experience survey scored a 39 out of 100 in the winter, the lowest score in any category, and the simplest chart we weren’t looking at was a straight line up, it was pipeline volume against compute capacity.

By May, the merge train was more than 30 minutes. It was not the long runtime that was the issue per se, but what happened when the train failed. Not only would an occasional failing test delay a single build but cause delays for all the builds behind that one in the queue, which would be canceled and have to be rerun from the start, although the change was unrelated to the failure.

Due to the out-of-memory errors and exit-137 errors on the node worker and the infrastructure not being able to handle the load properly, about 4,000 runs were canceled in May alone. The figures were obvious.  About a quarter of the runs finished green. Another quarter started to fail because of unstable infrastructure, some tests started to be flaky again because of it. The rest were canceled because of being queued up behind a failing build.

The second recovery: three engineers, six weeks, reduced costs

The next improvement resulted from reworking the way we were using the system all along. We built upon the foundation that was already laid out across the organization and took three focused steps using our three-person engineering platform team in six weeks.

The first step that needed to be done was that most of the end-to-end tests, i.e. the slower ones where the product is tested the way customers would test them, were no longer part of the merge train at all.

This is what we established as the basis last quarter, with the product teams splitting their end-to-end tests into two types – the critical ones, which comprised about 15% of all the end-to-end tests and were responsible for the scenarios that we simply could not afford to break, and the deprecated ones.

The deprecated tests were slowly replaced by backend and component tests as the test coverage of our backend reached 80%. Meanwhile, the critical tests remained mandatory for the merge train, whereas the deprecated ones were moved to a schedule of running every two hours on the main branch. We had dashboards to check if the critical tests caught bugs before code hit the production.  

The effect was felt immediately, with an average runtime of the merge train dropping from about 31 minutes to 18 minutes, which is a 42% decrease, and with the ratio of successful merges per failed/canceled ones improving from one-to-three to 25-to-one on the first day. The tradeoff was also quite obvious – any failure of one of the offloaded tests resulted in blocking the deployment until sign-off from the corresponding team.

We took another look at how the tests were distributed in parallel groups. Rather than having them equally distributed, we leveraged the production timing data to balance out the distribution and cut the time needed for the longest group of tests by about a third, getting some to 15-17 minutes. Additionally, we have improved our fail-fast strategy and now a train run automatically kills itself a minute after it fails its first job (before that it was up to 70 test jobs).

As I’m writing this, the train is running at 15-16 minutes a pop for the last week, and roughly 95% of runs get completed green. It can now handle 130 merges a day or 22 an hour. The median time it takes from hitting merge until reaching the main branch is under 20 minutes. 

During the worst weeks of May, it would take half a day to complete the same run. Flaky-test loop, meanwhile, is now 100% automated – detect, classify, ticket, and auto-skip – but it is still capturing 20 flaky events a day because flakiness is ‘weather’ rather than a bug that can be fixed once.

The bill got smaller too; the 75th percentile run time – the time three out of four runs beat – was reduced from 33 minutes to 20, while CI runner costs were reduced by 40%. Meanwhile, cancelation storms had turned the merge train pool into our biggest CI line item – more than all of our tests machines combined – and reduced its cost by 70%.

The future continuous integration pipeline

Do the math that started off the post with today’s figures, around 60 plus merges per day and 700 runs through the pipeline per day, and the value of one minute of train time is even more than it was in 2025. The mathematics still holds true, but we had just stopped using it.

That is what I will do anywhere else because I will treat pipeline time as a product metric and allocate a platform team and/or a product manager who is in charge of meeting a target and a budget. That is what we currently do. This team is responsible for a 75th percentile 15 minutes target in Q3, and ten minutes target by the end of the year.

Do you have an owner and a number for next quarter or just another happy story from last year?

Get articles like this in your inbox

Choose your LeadDev newsletters to subscribe to.