CircleCI announced the findings of 2020 State of Software Delivery

CircleCI announced the findings of a new report, 2020 State of Software Delivery: Data-Backed Benchmarks for Engineering Teams.

“As the world’s largest standalone CI provider, we have a unique opportunity to investigate what software delivery looks like quantitatively: across tens of thousands of teams, commit by commit. So, we looked to data from 11 million workflows on our platform to see how teams were building and deploying software in practice to answer this question once and for all: What does a high performing team really look like?,” said Jim Rose, CEO, CircleCI.

While many of the report findings point to an increase in high performance among engineering teams, especially around when the global pandemic began, the report also uncovers some improvements that can be made.

Key findings include:

Shortly after COVID-19 shut down our economy, developers stepped up. During the month of April 2020, developers pushed more code than during any other month this year and also had the highest success rate. The increase in code was also observed over the weekends. When success rates lift overall, it indicates that teams are taking a step back from innovation and focusing on shoring up their business-critical systems.

Most teams aren’t deploying dozens of times a day. Year after year, we hear in the industry that high performing teams are deploying dozens of times per day. While those stats make for great conference talks, our data doesn’t corroborate this metric. Fifty-percent of workflows in the data set were run less often than once per day, at 0.7 times per day. In 2020, developers moved slower, tested more, and failed less as a result. The longest recovery times recorded in 2020 are shorter than those in 2019, signaling that workflows took longer to run. This was due to more testing, which resulted in less failure. With fewer failed builds the overall velocity of organizations increased compared to 2019.

Globally distributed organizations have a significant software development advantage. How fast companies can fix their mistakes and recover is the key to winning. Half of all organizations resolved their failed builds in under one hour. CircleCI’s hypothesis has to do with distribution. For instance, an organization with developers all located in one town working typical business hours may not be able to resolve failed builds before leaving for home. At the same time, an organization with a global development team likely has processes in place to hand off work between teams, eliminating this daily delay.

Controversial branch naming did not change despite the Black Lives Matter movement. We saw a lot of media attention and social media chatter earlier this summer around the naming that teams use as the default branch on their VCSs. The discussion revolved around the renaming of the default branch of projects from ‘master’ to ‘main’ or another option. Despite widespread agreement and intentions, the data doesn’t indicate much movement from organizations renaming their default branches away from ‘master’.

“If 2020 has shown as anything it's that people need to be able to work in ways where they feel comfortable, with reasonable expectations and support in order to avoid burnout. One really useful aspect of the report’s findings then is a focus in setting reasonable key metrics and performance benchmarks,” said James Governor, analyst and Co-founder, RedMonk.

The first-ever benchmarks that CircleCI identified for teams practicing CI/CD include:

• Throughput: the number of workflow runs matters less than being at a deploy-ready state most or all of the time.
   
• Duration: teams want to aim for workflow durations in the range of five to ten minutes.
   
• Mean time to recovery: teams should aim to recover from any failed runs by fixing or reverting in under an hour.
   
• Success rate: success rates above 90% should be your standard for the default branch of an application.

Additional findings identify trends in development velocity and call on leaders to prioritize developer empathy and team resilience as ways to avoid potential burnout.