Continuous Delivery Drives Business Advantage: Six Leading Edge Companies

Several years ago, the application lifecycle process known as continuous delivery was a new concept being tried by a small group of early adopters. Things have changed. Now, continuous delivery, or CD, is a practice that companies big and small are not only trying out, but embracing as part of a new, faster moving, on-demand business culture. 

We’re now hearing the term described often as being a “table stakes” necessity that companies need to pursue to compete in their particular field. Companies that ante up have an advantage. Companies that don’t, fall behind. Continuous Delivery is now seen as the first stepping-stone of a DevOps transformation.

Here are some examples of how several prominent companies are using CD for business advantage.

You’re not going to find many advertising and data platforms that are busier and more complex than the Yahoo! Ad Exchange. The media titan’s platforms are backed by a massively distributed system that processes over 100 billion events each day. The system consists of hundreds of unique software components and thousands of servers. Hundreds of programmers are working in more than a dozen languages, on different teams with different priorities and schedules.

Today, the system hums – thanks to a process overhaul driven by a move to continuous delivery. The Ad Exchange team generates more than 8,000 builds a day, committing code to production without human intervention.

Before the CD initiative, it was a different story. The Ad Exchange team launched builds to production only three times per year. Builds broke regularly, and they’d take a week or so on average to rebuild. Coding would take place in intense six-week sessions, followed by eight weeks of testing. QA would sign off with a long list of exceptions and it would take another four weeks to launch into production.

“The same team today launches regularly,” Yahoo! senior product architect Stas Zvinyatsokovsky told the audience at his 2014 Jenkins User Conference session in Palo Alto. The team goes from commit to certification in about six hours and they queue up to launch every day. If there’s a break or a security issue, they fix it the same day. 

Zvinyatsokovsky says the Ad Exchange team has gone “from ‘continuous debacle’ to continuous delivery” in the space of two years.

“Continuous delivery has become part of the culture,” he said. “Nowadays, when a new product is delivered we expect then to launch to production multiple times a day.”

Etsy has generated plenty of buzz in marketing itself as a cutting-edge online destination for handmade goods, vintage items and craft supplies. Style.com, for one, went so far as to say in December 2014 that the shopping site is “ready to take over the world of retail.” 

Etsy also has emerged as a trend-setter in the world of continuous delivery. The way company officials describe it, Etsy had to amp up its investments in CD to keep up with business demand. “Our site is so successful,” Senior Software Engineer Ross Snyder explained at the 2013 SXSW show, the company had to determine “how can we move fast enough to keep up with demand.”

These days, Etsy moves fast. The company deploys API changes in 18 seconds and launches a new website every 150 seconds. The company does about 50 deploys a day, according to a presentation at the 2014 QCon. That number is about double what Etsy was doing in 2012, when 196 different people deployed to production and when the retail-heavy months of November and December were swelling to more than 20 percent higher than average volumes.

The key was to commit to making many small, continuous changes. To be comfortable deploying changes, Etsy adopted a range of tools and practices: mandatory IRC-based communication; developer virtual machines; continuous integration; one-click deployments; thorough application and system monitoring; no blame post-mortems and on-call policies for both dev and ops teams, according to an article in InfoQ.

Along with improving productivity, the move to continuous delivery has generated an added benefit. Said Snyder, at SXSW, it has helped Etsy attract engineers “who like to move fast and get stuff done.” 

Choose Digital provides a white-label digital content marketplace for Marriott Hotels, United Airlines and other enterprise partners. To continually update its content and user features, the company adopted a development and deployment platform leveraging Jenkins CI, an open source continuous delivery platform.

“Repeatable automation – that’s where you’re going to get the real benefits for your business,” Choose Digital CEO Mario Cruz explained at the 2014 Jenkins User Conference in San Francisco. “The sooner you can automate, the sooner you can get the time back to innovate on your code…I want my developers creating money for the company. You can’t innovate if you’re stuck in the technical debt piece.”

Using Jenkins, Cruz set up a system where builds get pushed to production immediately, automatically, and then stopped for a series of rigorous tests. “The biggest secret to development is running tests,” he said. “Tests are your friend. You should spend more time on tests than anything else. But, when all tests are passed, it is auto-deployed. It’s that simple.”

Amazon 

What can a person accomplish in 11.7 seconds? A fast runner could sprint a little more than 100 yards. A nimble texter could connect with several friends in rapid-fire fashion.

Amazon engineers can push out a new software update in about 11.7 seconds. They do that all week long – one after another, five times a minute, 300+ times an hour.

Amazon has improved its delivery processes through a pair of strategic moves: shifting from physical to virtual servers and adopting a continuous deployment process that runs on an internal system Amazon calls Apollo. The Apollo system allows any developer to deploy his own code whenever he wants, and to whatever fleet of servers he needs. “It is one of the most important pieces of special sauce that we have,” John Jenkins, former lead engineer at Amazon, told blogger George Lawton in a 2013 piece in ServiceVirtualization.com.

 

The two key elements of the Apollo system include the concepts of environments and stages. Stages allow environments to be shifted to different resources for development, testing and production.

Though these numbers date back a few years, Amazon’s experience illustrates the power of CD in action. In one slice of time – May 2011 – Amazon kicked off a new software deployment to production every 11.6 seconds. In the busiest hour, 1,079 new deployments shifted to production. The mean and maximum number of hosts simultaneously receiving a deployment exceeded 10,000, and 30,000, respectively.

Neustar’s business focuses on providing back-end mission-critical services for operations that need to be delivered with a high degree of reliability. Things like operating a cloud-based digital rights library, administrating top-level Internet domains and providing managed DNS services for businesses across the globe.  

To deliver software quickly and reliably, Neustar adopted a continuous delivery approach utilizing a cloud-based development platform based on Jenkins CI. “CI with Jenkins helps us meet our time-to-market objectives because it empowers our development teams to build and deliver software as quickly as they can,” according to Jason Shawn, director of cloud services and platforms at Neustar.

Neustar’s move to CD aimed to shorten the development lifecycle, increase build frequency and minimize IT overhead. It worked. Following the Jenkins implementation, the company cut infrastructure maintenance overhead by 66 percent, reduced development time by 25 percent and increased the build frequency by a factor of 4.

“We had a number of jobs that we used to run nightly that we now run three or four times daily,” Shawn said. “Other jobs that we ran only on demand, we are now running nightly. In the end that kind of improvement enables us to move from feature request to deployment faster, and that is what matters to our customers and our business.”

Orbitz Worldwide, the large global online travel company, needed a mechanism to streamline an extensive and disparate software development process consisting of more than 350 developers on 35 different teams. These teams oversee more than 180 applications that power 13 different websites.

To shorten delivery times, automate tests, ensure reliable builds and increase build consistency across the organization, Orbitz implemented a continuous delivery process based on Jenkins CI.

The results were encouraging. Release cycles were cut by more than 75 percent, teams have shifted their focus to higher-value tasks, and Orbitz customers’ user experiences were enhanced through an increase in multivariate testing.

“Before we refined our software delivery processes, it would take more than two weeks to cook test results into the main website and move on to the next level of testing,” said Jacob Tomaw, principal engineer at Orbitz. “Now we can push updates made based upon test results into production within a few days.”

In the past, patches were the only way to get code into production outside of the two-week release process. Today, significantly fewer patch requests are made and most of them are business-driven, originating from product owners rather than from developers. 

With faster cycle times, Orbitz instituted a new policy that requires each patch to be evaluated for its contribution to margin. In order for a patch to be approved, it has to contribute projected revenue at least equal to the labor expense of deploying it - otherwise it must wait for the next release, which is now at most four days away instead of 18.

A decade ago, continuous delivery was seen as a new and confusing concept – something for early adopters to test and play around with while the rest of the business community stuck to their cycles. Nowadays, many companies are still figuring out the best ways to integrate CD into their operations. But CD is making its way into the mainstream. 

Companies small and large – from Choose Digital all the way up to Amazon – across industries as varied as retail, travel and digital music delivery are generating real, tangible results. They’re deploying more often, with less friction. They’re creating whole new business models and they are beginning their DevOps transformation – thanks to continuous delivery.