LinkedIn recently released Project Voyager, their codename for the new version of the company’s flagship application for Android, iOS, and the mobile web catering to its over 400 million users. According to the LinkedIn dev team, Voyager is the result of more than a year of product development work by over 250 engineers. They rethought the LinkedIn experience from the ground up, merging the product perspective and engineering side.
Incredibly, with the launch in the new app, the LinkedIn dev team established a 3x3 release cycle – releasing three hours a day with no more than three hours between when code is committed and when that code is available to users. Now that’s continuous delivery on steroids!
Here is what Drew Hannay with the LinkedIn dev team had to say about their rationale in a recent blog post:
Before Voyager, we released the LinkedIn app once per month. Prior to each release, we picked a release candidate and handed it off to the testing team, which performed a manual, four-day regression test suite. If they found any bugs, we made a hotfix and gave the testing team a new build. Depending on the severity of the bug, this might require another round of testing. Engineers rushed to get their code checked in before the monthly release candidate deadline, or waited an entire month to deliver their features or bug fixes to our members.
Product managers and marketing partners had to plan their schedules around this release process, and hope that each release went out on time. It wasn’t easy to iterate on member feedback because we had only 12 releases a year. This setup wasn’t ideal for anyone. We wanted the release cadence to be a product decision, not something determined by engineering and testing constraints.
The two main reasons he cites for the 3x3 goal are these:
1) Three hours is not enough time to conduct any manual testing steps, so holding to this constraint ensures they won’t revert to using manual validation to certify their releases.
2) Three hours is not enough time to test everything end-to-end. As Hannay comments, “This may seem counter-intuitive: the more tests we have the better, right? But remember that the goal is not just to automate our releases, but also to enable us to iterate faster. If engineers spend 20% of their time writing new code and the rest of their time refactoring the fragile UI tests that break every time code is changed, we aren't achieving our goal. We prefer an approach where product owners decide which production-critical paths require UI tests, and resolve some of the edge/negative cases with unit tests, which are much faster and easier to maintain.”
He then discusses the establishment of a complete automation pipeline for every step, from code commit to production release, how they worked with partner teams (such as a localization team would confirm that translations of new features), and other challenges with a 3x3 release cycle.