Why software is no longer being written from scratch

ADM: What is a software supply chain?

Software supply chains are just like supply chains in other manufacturing entities used around the world. A typical supply chain has buyers who build parts and make those available to manufacturers through a number of distribution channels. Those manufacturers get those parts and use them to assemble finished goods that they then sell to their customers. In software supply chains, the parallels are very common to traditional supply chains but the suppliers are open source projects that create open source and third-party components. Those components are then made available on the internet through large public warehouses of open source components.

Any software development team around the world that is manufacturing software using these parts as building blocks to assemble their own applications can freely access these warehouses. These components are then assembled through the software development teams into finished goods, which are software applications that all of us either rely on for services or as end products that we're purchasing.

ADM: Why is software no longer written from scratch?

I think a lot of people who are not familiar with how development has changed in the last decade believe that software is built from scratch, and that there are developers out there who code every single line within an application.

In reality, use of open source and third-party components over the past 10 years has become a commonplace development practice. Developers are sharing their best code by packaging it up into components for other developers to reuse. So, rather than write my own logging framework, web application framework, or encryption functions for an application, I can actually go to the internet and source those for free from developer’s open source projects that have supplied the parts. What this means is that as a developer I don't need to write from scratch anymore and I can accelerate the pace of developing new applications. The proliferation of open source has added a tremendous new velocity to software development practices around the globe.

ADM: How many open source components are being consumed, and are all parts created equal?

Open source components are being consumed at almost unimaginable volumes today. In the Java realm of software development last year, we saw more than 31 billion download requests happen across a global population of about 10 million Java developers. While Java developers are consuming billions of these component parts, component use is not limited to Java development alone. There are different component formats for different development languages - component formats like npm for JavaScript, NuGet for .NET developers, PyPI packages for Python developers, etc.

Within these billions of components, one of the secrets out there is that not all of these parts are created equal. There are millions of parts available to developers. Of those millions, versions could be as young as one day or as old as 11 years. The average open source project releases about 14 new versions of their component parts per year. Some of those new releases are to make the components higher performing, less buggy, more functional, or more secure.

Last year in the research that we did,we saw that more than six percent of the components that were being downloaded had a known security vulnerability. Across billions of downloads, about 1 in 16 components had a known security defect on the day they were downloaded.

ADM: How are organizations vetting the quality and security of components in their software supply chains today?

The evolution of software development requires the need for more DevOps-native tools that allow developers to automatically evaluate millions of components. An example I like to use is that of a big healthcare company with 2,000 software developers, that’s consuming 16million parts annually. There's only one person employed at that business to assess whether those parts are good or not, and that person only looks at the software licenses of those components. That one person doesn’t look at the version, how old the components are, or any known security defects. Additionally, that one person alone cannot keep up with annually auditing the 16 million parts being consumed by their developers.

The day-to-day reality inside that organization is that the person in charge of approving components is busy 100%of the time. Even if they could employ 100 more people in that role, the organization could not keep pace by manually evaluating what they are consuming.

ADM: Are existing practices to identify and track defects in open source components keeping pace with development?

The existing practices in the industry today are really more manual than automated and when manual practices are in place, those practices cannot keep up with the volume of activity that is happening across the industry. An average organization consumes about 225,000 components a year in the manufacturing or development of its software applications. For an organization to try and evaluate every one of those components and determine whether it is good or has a known defect is very difficult and time-consuming.

In fact, the volume of consumption has out-paced manual approaches to evaluation and governance of these components for probably six or eight years now. Many companies have chosen to approach automated ways to identify, track and trade, and set policies for which components are acceptable to use in their organization and which are not.

For example, a company I know in the financial services industry has a governance practice in place around defining which components their developers can use. They told me that they had more than 800 components approved across their application development portfolio. However, when we worked with them to analyze how many components were actually used in their portfolio, they found that developers were actively using 13,000 different open-source and third party components.

ADM: Can you share an example of where open source governance practices are not keeping pace?

Part of the discrepancy between the number of components that were approved (800) and the number that were actually in use (13,000) had to do with the approval or governance process that was in place. What developers wanted to immediately know was which components were safe or unsafe to use, or which fit or did not fit with the company's policy. As a result they were forced to wait anywhere from two to nine weeks for a response from the governance body within their company. When developers want an instant decision and don't get one they will find a workaround. This workaround led to more than 12,000 components being actively used in development, skirting the approval process.

This workaround led to lower-quality components being used actively throughout the development organization, and that's something that they're now asserting more control over while allowing their developers to use the highest-quality components permissible from open-source projects worldwide.

ADM: What lessons can we learn from traditional manufacturing to improve how software is developed?

There are lessons from traditional manufacturing practices, especially high velocity, high volume manufacturing that can be applied to software. Many of these lessons originate from those learned at Toyota through Deming and other manufacturing thought leaders. The key practices that I've seen employed through a number of organizations relate to relying on the fewest and highest quality suppliers. The leading organizations are also tracking and tracing use of components across their software supply chains; in a situation where any of them are discovered to be defective, you can immediately locate those components and begin to quickly remediate them within the organization.

In the 2016 State of the Software Supply Chain report, one of the common practices that we saw being applied from traditional manufacturing into software development was the use of a software Bill of Materials. We highlighted how organizations like Exxon and the Mayo Clinic were using a software bill of materials to identify what components they were using in the applications they were developing in-house, as well as applications that they were purchasing from other development organizations. They were using these software bill of materials to determine what component parts were used in those applications because they wanted to understand if any of them had a known defect -- and in particular did they have any known security defect.

ADM: If open source software components have so many defects, should we stop using them?

Most organizations that hear how many open source and third-party components they're consuming and also understand the defect rates, have an initial reaction of believing they need to stop using these components. They get scared and feel that they can’t allow that volume of risk to come into their organization.

However, the reality is software development practices rely so much on these components that most organizations that would make the decision to stop using components would simply have to stop developing software. The use of components has proliferated so much it's nearly impossible to stop the consumption.

Given that scenario, it's important that we learn to manage software supply chains, just as other manufacturing organizations have learned to manage their supply chains. Organizations can do this in a high volume, high velocity environment while maintaining quality. If we think about manufacturing organizations around the world, like an Apple, a Ford Motor Company or a Pfizer…they are using a huge number of parts to assemble or produce the goods that they are then delivering to customers.

These companies have figured out how to work with their suppliers and their supply chains to vet the components before they come in the door. They use the highest quality and latest versions of parts in there in order to deliver the best products to their customers. By managing these supply chains they've proven they can reduce the cost of producing goods by using the highest quality parts.

ADM: Have organizations quantified the value of better managing their software supply chains?

Absolutely. We have worked with a variety of different organizations that have utilized DevOps-native tools to reduce the number of defective components they use. This has been done by using high quality components and reducing the variety of components across their software supply chain. We've seen organizations not only increase or improve the quality of their software application by as much as 50 or 60 percent, but we've also seen them improve the productivity of their development organization by as much as 30 to 40 percent in the same time frame.

Organizations can see the same type of results by using the highest quality parts from the start, and as a result those development organizations spend less time and work fixing these defects that may be caught later in the software development lifecycle or even out in production environments.

Organizations that choose to manage their software supply chain from the earliest stages and bring in the highest quality parts can reduce the amount of time, effort, and money that they're spending to remediate these defects. They then can apply that money towards their innovation budget to continue to differentiate their businesses and make them more competitive.