Driving Software Recalls: Manufacturing Supply Chain Best Practices for Open Source Consumption

Executive Summary

In December 2021, the Log4j vulnerability, Log4shell, crippled development teams worldwide. The exploit itself was not what dragged teams down. Instead, most software organizations could not identify whether and where they were using Log4j. This meant developers needed to review entire codebases to determine their exposure and risk. For large enterprises with thousands or tens of thousands of applications, work on new features came to a halt.

Log4shell was another example of software organizations failing to acknowledge or recognize (likely both) that open source software (OSS) is more than just a technological innovation—OSS wholly changed how software products are created. Over more than two decades, OSS catalyzed an already growing movement towards componentized software development—where applications are developed in parts by different internal and external teams. In many ways, OSS transformed the industry into something that more closely mirrors traditional manufacturing.

While there is no 1:1 comparison between software development and other forms of manufacturing, there are still many similarities that provide a learning opportunity. Specifically, by looking at automotive manufacturing, there are modern supply chain management best practices capable of improving OSS consumption and software supply chain security. These same mechanisms can also improve the processes software manufacturers use to disclose the presence of vulnerabilities to their customers.

It is the latter point that is most critical. Like expectations set for any other manufacturer, customers expect software manufacturers to follow a standard of care to ensure their products are safe and secure by design. More importantly, if there is a defect in a product, customers expect a manufacturer to communicate and remediate the defect.

Traditionally, this process of notification and remediation is referred to as a recall. Like comparing software and automotive manufacturing, a recall process for software cannot be applied exactly. Yet, there are similarities. Critical elements of recall processes could provide a mechanism to hold software manufacturers accountable for the parts they use and for communication of critical vulnerabilities (defects) to their customers. However, many manufacturers do not, or cannot, track the OSS used in their software products. Worse yet, in many cases, they lack awareness of critical vulnerabilities in the software products they manufacture.

This paper aims to demonstrate how principles from modern automotive manufacturing, specifically those from W. Edwards Deming, a leader in supply chain and management theory, can be applied to improve OSS consumption and supply chain security. With these processes in place, software manufacturers can minimize the impact of vulnerable OSS and communicate to customers when those defects are encountered. To implement these improvements, policy from the federal government will need to provide further guidance, direction, and accountability.

For software manufacturers, this means:

Building security into software products by design.

Consuming only high-value OSS components and projects.

Continuously tracking, monitoring, and improving OSS consumption.

For policymakers, this means:

Holding software manufacturers responsible and accountable via a national standard of care.

Requiring software manufacturers to demonstrate their approach to vetting OSS used in their products.

Driving software manufacturers to continuously track, monitor, and improve OSS consumption.

Introduction

On December 10, 2021, around three weeks before much of the world logged off for winter and end-of-year holiday festivities, arguably the worst software vulnerability ever discovered, Log4shell,¹ was publicly disclosed. And this is not something said lightly. Following heavy-hitting vulnerabilities like Heartbleed and Shellshock, Log4shell had an unprecedented impact.² The vulnerability affected Log4j, a ubiquitous open source logging framework used to track information and errors in computer systems.³

Logging tools provide critical functionality to software organizations today, helping enterprises investigate and determine causes of unexpected operations of everything from websites to applications on your phone. For example, if a server suddenly shuts down, those logs help pinpoint the root cause. Log4j, which the Apache Software Foundation manages, is used in almost every Java application,⁴ especially at the enterprise level.⁵ Log4shell was critical both because it was easy to exploit and due to its potential widespread impact, which included servers providing critical access to secured networks and sensitive data at the private, commercial, and national levels.

The US Cybersecurity and Infrastructure Security Agency (CISA) has cited the scale of the Log4shell vulnerability across much of their published best practices. In July 2022, the first report from the Cyber Safety Review Board (CSRB) provided updates on lessons learned from Log4shell as well.⁶ And then in March 2023, the National Cybersecurity Strategy stressed the importance of open source software (OSS) security and its impact on supply chains. Unfortunately, with the historical absence of meaningful cybersecurity regulatory oversight, organizations and individuals must often voluntarily adopt these best practices and recommendations, especially in engagements outside government activities.⁷ This gap is punctuated by evidence indicating this is not happening across the board.

In 2022, a year after the disclosure of Log4shell, a study of current Log4j downloads indicated that as much as 30 percent of users were still using vulnerable versions.⁸ Some of these cases were potentially due to hubris or lack of care. However, the more likely cause of continued downloads of vulnerable versions of Log4j was an organization’s lack of visibility into the OSS they consume. Without this insight, organizations cannot effectively respond to vulnerabilities, including communication of the presence and effect of those vulnerabilities on users (customers) of their software.

There is a general expectation that products like food, vehicles, and other goods should be inherently safe. In cases where products include components known to be harmful or defective, manufacturers have a responsibility to disclose and remediate that risk through a recall process. When this is impossible, manufacturers are usually obligated by regulatory policy to warn their customers of the potential danger of defective products. Yet, software products are not held to the same standard; this must change.

When defects are present, like their peers, software manufacturers are responsible for communicating potential risks to their users and guiding them through remediation options. Fortunately, addressing the awareness of OSS consumption and improving communication related to OSS vulnerability disclosure does not require every aspect of a typical recall process deployed in manufacturing. However, achieving those improvements does necessitate software manufacturers track and monitor all the OSS they consume and incorporate into their software products.

Starting with an analysis of the Log4j vulnerability and the corresponding response by software manufacturers, this paper aims to provide a better understanding of OSS consumption, the role OSS plays in the modern software supply chain, and relevant parallels to traditional manufacturing, specifically in the automotive sector. This comparison provides an opportunity to borrow essential mechanisms tested across many years against similar challenges. Looking specifically at automotive manufacturing also provides an opportunity to isolate the best and most relevant examples, especially those pioneered by W. Edwards Deming, a pivotal figure in modern supply chain management. By building awareness of OSS consumption, software manufacturers can improve their ability to effectively respond to issues like Log4shell and facilitate risk communication through existing coordinated vulnerability disclosure (CVD) practices, such as publishing advisories and notifications, that closely resemble recall processes in manufacturing. While disclosure and communication are critically important, this paper’s primary intent is to assert software manufacturers’ responsibility to continuously track, monitor, and improve their consumption of OSS at an organizational level.

The Severity of Log4shell

Log4shell (CVE-2021-44228)⁹ earned the highest Common Vulnerability Scoring System (CVSS) score, level 10 (critical), in its official Common Vulnerability Enumeration (CVE) disclosure.¹⁰^, ¹¹ For perspective, an independent database of CVSS scores shows just 4 percent of all CVEs ever recorded (over 200,000 through twenty-three years of reporting) received a score of ten, which is typically limited to high-impact vulnerabilities that are also easy to exploit.¹²

In the case of Log4shell, the vulnerability allowed remote code execution—the ability for bad actors to remotely make changes to, run software on, and take control of a system. While this type of exploit is terrible in any situation, what made Log4shell so potentially dangerous was its ubiquity within the Java ecosystem. In an article from Wired magazine shortly after the official disclosure, Log4shell was characterized as something that would “haunt the Internet for years.”¹³ Echoing that sentiment, just four days after the Log4shell disclosure, CISA Director Jen Easterly briefed industry leaders on the situation, saying, “[the exploit] is one of the most serious I’ve seen in my entire career, if not the most serious.”¹⁴ Jay Gazlay of CISA’s Vulnerability Management Office followed Easterly’s comments, stating, “Hundreds of millions of devices are likely to be affected.” However, that number is likely low, given estimates of the breadth of affected companies and projections from experts that the vulnerability will persist for years to come.¹⁵

By January 2022, there were already multiple reported examples of bad actors exploiting the Log4shell vulnerability.¹⁶ By September of the same year, the US government published an advisory confirming that the Federal Civilian Executive Branch (FCEB) had been compromised.¹⁷ Considering the Log4shell vulnerability was present in versions of Log4j since 2013, there is a high likelihood that attacks took place for some time before the official disclosure.

Unfortunately, exploitation of a vulnerability does not necessarily mean software manufacturers will pay attention. CISA’s Cybersecurity Strategic Plan FY 2024-2026 highlights that “most intrusions today are perpetrated using known vulnerabilities or exploiting weak security controls.”¹⁸ A telemetry analysis from Tenable, a cybersecurity risk firm, found that 72 percent of organizations were still vulnerable to Log4shell as of October 2022.¹⁹ To understand how that is possible, when this article was written, a review of Maven Central, the largest repository of open source Java components, showed that 23 percent of new downloads each week represented versions vulnerable to Log4shell.²⁰ This equates to hundreds of thousands of vulnerable versions of Log4j entering software supply chains every month.

However, the negativity surrounding the Log4shell vulnerability is only part of the story. The spotlight on Log4j demonstrates OSS’s tremendous impact on modern software development. If software manufacturers are unaware of the vulnerability of critical OSS like Log4j, what about all the other OSS they consume?

The Problem is Not Open Source Software

Today, almost 90 percent of modern applications are composed of OSS, including components like the Log4j logging framework.²¹ On average, about 11 percent of those OSS components have known vulnerabilities. While the extensive use of OSS has reduced the cost of research and development, catalyzed incredible leaps in innovation, and drastically decreased the time it takes to deliver critical business functionality, it has also created a security conundrum.

Unlike other third-party software, in most cases, OSS is not “supplied;” that is, projects or individual developers that maintain the software do not act as suppliers. As described in Chinmayi Sharma’s “Tragedy of the Digital Commons,” OSS is a “public good” or a natural resource freely available to anyone.²² As a result, its creators cannot know how their work will be used or, in many cases, how software manufacturers may modify it to fit their customers’ needs. Instead, it is the responsibility of the software manufacturer to ensure that any final product delivered to a customer that uses OSS is free of defects (i.e. vulnerabilities). This relationship differs from traditional supplier engagement and procurement processes used to offset research and development costs. This difference presents considerable potential for risk.

A componentized approach to software development is not new. Many organizations still outsource the development of specific frameworks and other elements of an application to third parties. However, these engagements use standardized procurement and review processes that ensure products match technical specifications and other contractual requirements. In contrast, for many organizations, the consumption of OSS has no equivalent process.

This lack of process is both a blessing and a curse. Development teams can use and modify whatever they find among the vast amount of available OSS. That freedom allows them to completely sidestep the procurement process used when working directly with third parties. This circumvention often is not done intentionally. In most cases, development teams simply do not associate OSS consumption with procurement. This lack of overhead can create short-term benefits for innovation. However, knowing where OSS is used is just as crucial as what OSS is consumed. Without common standards, software manufacturers often do not track their OSS consumption, making it extremely difficult to monitor and identify defects.

By ignoring OSS consumption, development teams expose the organization to increased risk, especially when vulnerabilities are discovered. This extends well beyond Log4shell: the same report that identified continued downloads of vulnerable versions of Log4j also found that 96 percent of OSS downloads with a vulnerability have non-vulnerable updates that are available.²³The lack of visibility into the consumption of OSS means software manufacturers often miss these fixes and patches that would reduce and, in most cases, entirely remove the risk associated with vulnerabilities in previous versions.

The Impact of Vulnerabilities

The term “vulnerabilities” sounds frightening; sometimes, they can have that potential. However, vulnerabilities generally are not an injection of malicious code but rather an inadvertent weakness in the code itself.²⁴ In many ways, vulnerabilities can be as simple as a typo in any written content. This broad definition is true of all code, open source or proprietary. The nature of a vulnerability changes when it can be exploited. In other words, when those bugs allow bad actors to access private systems, the vulnerability represents risk.

The presence of risk does not mean a vulnerability is inherently critical to every organization and every software product. Depending on the context, vulnerabilities may present little to no risk, even in the case of a critical OSS component like Log4j. However, it is impossible to understand and address a risk without a clear understanding of OSS consumption, implementation, and configuration. This lack of understanding directly impedes software manufacturers’ ability to respond to critical issues effectively.

Once again, using the Log4j vulnerability as an example, much of the time development teams spent addressing Log4shell was not focused on implementing fixes and applying patches to Log4j. Instead, the lion’s share of initial investment was spent trying to fully understand their use of Log4j in the first place. So, before the arduous technical work could begin, teams first needed to figure out their exact version of Log4j and where the vulnerable versions existed across their portfolio of software products.²⁵. Again, all this must happen before any fix or patch can be applied. This can quickly become an impossible task at scale for large organizations with a complex code base and tens of thousands of applications.

Regardless of organizational size, the reactive nature of a software manufacturer’s response to vulnerabilities is the best example of the current weaknesses in OSS consumption management. Yet, that weakness is almost entirely avoidable. If software organizations track what OSS they consume, where that OSS is used, and then monitor OSS for defects and other quality parameters, their response can be far more proactive. In many cases, issues can be addressed long before a product ships. Even when that is not possible, the ability to triage and prioritize remediation efforts avoids chaos when a vulnerability is discovered, allowing teams to take direct control of the response and tackle defects strategically.

The inability of many development teams to effectively respond to Log4shell should be a call for software manufacturers to change. At the center of this change is the acceptance and adoption of processes that acknowledge OSS is not simply a way to bypass traditional procurement. Instead, OSS must be a critical consideration in managing a software supply chain. Achieving a paradigm shift like this requires tested principles and mechanisms. Luckily, there are a plethora of modern supply chain management best practices that can be borrowed from other manufacturing industries, especially automotive manufacturing.

The Advantage of Modern Supply Chain Management Principles

While nuances exist, the intent of this paper is not to draw a direct line between software development and automotive manufacturing. Instead, it is to compare processes in both industries, especially related to supply chain best practices. This is also not to say that manufacturing has everything figured out. Even in recent history, there have been low points, such as the combined failure of General Motors and the National Highway and Transportation Safety Administration (NHTSA) to recall faulty ignition switches in the Chevy Cobalt.²⁶ However, despite these setbacks and continued opportunities to improve, automobile manufacturers have developed efficient and effective processes for identifying and communicating defective products. Through targeted notifications and safety recalls, automotive manufacturers collectively communicate defects for millions of vehicles each year.²⁷

In many cases, recalls are related to discovering and communicating severe safety issues that could cause serious injury and, in some cases, death. In this way, the volume of recalls represents drastic improvements to consumer safety.²⁸However, a common misconception is that recalls are a way to pull a defective product back. While this works in some cases, for example, if a vehicle has not yet been sold, most recalls affect vehicles already on the road. This means the manufacturer must be able to identify not only defective parts but also the location of the affected vehicles. The critical point here is that recalls would be impossible without the ability to track parts in a vehicle throughout the supply chain and up to final assembly. Put another way, this ability to track and monitor parts means that when a defect is identified, the manufacturer can target their communication and any remediation steps to the affected consumers.

Of course, tracking and responding to defects is only a part of modern supply chain management. Manufacturers must also work to minimize defects, and this is where modern supply chain theory provides the most relevant and helpful guidance for software supply chains. Specifically, today’s software manufacturers should look to the work of W. Edwards Deming, who was responsible for helping rebuild automotive manufacturing in post-World War II Japan and was highly influential in the global automotive market. Most notably, Deming focused on improving supply chain practices and, more importantly, ensuring greater control over quality and security.²⁹

Deming insisted that manufacturers source the best parts from the best suppliers, which was at the heart of his strategy. As part of his recommendations, he put together a fourteen-point approach to quality management to support this endeavor.³⁰ Many of these ideas are now accepted concepts in manufacturing, like the Andon principle, which states that any worker should be able to immediately stop production to prevent defects and further quality issues down the line. While the complete set of fourteen principles dives deeper into management philosophy and is outside the scope of this paper, for software supply chains and improvements to OSS consumption, three are critical:

Principle 3: Cease dependence on inspection to achieve quality.³¹ In this principle, Deming suggests manufacturers “shift left.” By moving inspection earlier in the production processes, defects are found when changes are much easier to make. Inspection of the final product should still happen but should not be the only or first inspection point.

Principle 4: Move toward a single supplier for any one item on a long-term relationship of loyalty and trust.³² In this principle, Deming suggests that complexity is introduced by utilizing multiple suppliers for the same part. By utilizing the single, best supplier and building a relationship with them, when defects enter the supply chain, you only need to focus on reaching a resolution with a single supplier versus tackling issues from several suppliers simultaneously.

Principle 5: Constantly improve production systems to improve quality and efficiency, and thus constantly decrease costs.³³ In this principle, Deming aligns with the philosophy that you cannot improve what you do not monitor, and you cannot monitor what you do not track.

It is important to consider that quality can be highly subjective, establishing a singular definition of high-quality OSS is unnecessary. Rather, Deming’s principles offer an approach for software manufacturers to develop better processes for the consumption of OSS, which will enhance its quality in the long term. Here is how translating Deming’s guidance to software supply chains looks:

Principle 1: Build security into software products by design.³⁴ Like physical products, software manufacturers should be responsible for ensuring their products are safe and secure. Within the context of liability, this responsibility is often associated with a duty of care. In addition to a duty of care, a manufacturer’s level of responsibility to ensure products are safe and secure is a standard of care. To align with the National Cybersecurity Strategy and, in turn, meet a reasonable duty and standard of care, security needs to be a critical part of software manufacturing from the start.³⁵ For example, assessing the security of OSS in a product only after it is released is too late. Instead, software manufacturers must take an active role in their consumption of OSS at every stage of the Software Development Life Cycle (SDLC).

Principle 2: Use only the best, actively supported OSS components and build relationships with those projects and developers. Selecting the best OSS means evaluating it against criteria like known vulnerabilities, age, and average remediation/update times, among others. When an OSS component meets those standards, manufacturers should utilize it exclusively to avoid duplication and reduce their overall attack surface (risk). Next, select stable, supported versions of OSS and vet projects to ensure they utilize recommended processes and best practices.³⁶ Finally, build partnerships with high-quality open source projects and invest back into those projects to accelerate innovation upstream and reduce future costs downstream.

Principle 3: Continuously track, monitor, and improve the security of OSS that is being consumed. Manufacturers should understand how and where they consume OSS spanning the entire SDLC to reduce their risk related to known vulnerabilities. Software manufacturers should also establish criteria and develop organizational policies to improve the consumption of OSS. While efforts may start small, research indicates³⁷ a combination of modern tooling and best practices provide scalable and organization-wide approaches that can be applied across all teams and products without increasing costs or reducing productivity.

The three principles discussed above should define every software manufacturer’s core strategy for OSS consumption and guide their approach to software supply chain security. At the same time, it is worth a word of caution that an overcorrection can occur. The removal of all vulnerabilities is not necessary—if such a state is even achievable. As mentioned previously, exploitability and vulnerability mean different things. Many vulnerabilities have little potential for harm.³⁸ However, when a critical vulnerability does exist, once again, the most applicable lessons for software manufacturers come from a set of mechanisms used by automotive manufacturers to identify and respond to defects.

The First Goal of a Recall

An automotive recall is usually, although not always, conducted in partnership with the National Highway Transportation and Safety Administration (NHTSA), which, among other activities, investigates defects in automotive products.³⁹ At the end of an investigation, the NHTSA provides a non-binding recommendation to the manufacturer on how to recall the product.⁴⁰ Though automotive manufacturers can go against the recommendation, the NHTSA can seek legal action to ensure vehicle safety standards are upheld. However, any automotive manufacturer will likely say that their first goal for recalls is to avoid them altogether.

While a recall represents improved safety, recalls also represent significant expenses. For example, one consulting firm—AlixPartners—found that 2016, recalls had cost the automotive industry $22.1 billion.⁴¹ Adding a bit more detail, Forbes cited that the average per-vehicle cost of a recall was about $500 over the last 10 years.⁴² However, the cost of some recalls is much higher—for example, 2021’s Hyundai recall of 82,000 vehicles, which came in at $11,000 per vehicle, sets a new benchmark.⁴³ Like any business, automotive manufacturers focus on minimizing costs. And in the case of a critical defect requiring a recall, costs can increase exponentially. So, to better mitigate and minimize costs due to defects, many automotive manufacturers utilize processes and best practices related to supply chain management like Deming’s principles. This approach allows automotive manufacturers to proactively address defects in production and respond to customers quickly, efficiently, and effectively.

For example, in 2020, Toyota encountered a potential coolant leak defect caused by a faulty water flow meter used to manufacture their engines.⁴⁴ In this instance, the supplier identified the issue with the water meter but found no evidence this had created a defect in the engines themselves (it is important to note that Toyota supplies its own engines). Because the supplier “found no abnormalities,” it continued to ship the engines for final assembly at a Toyota manufacturing plant. However, according to Toyota’s investigation, the coolant leak defect was detected in vehicles awaiting delivery to dealerships, as well as a small number already in dealer inventory. Luckily, Toyota could use serial numbers from the defective engines to trace the leaks back to engines manufactured using the defective water flow meter over a three-month period in 2019. With that information, Toyota conducted an official recall, communicating with dealerships and customers to coordinate inspections and repairs.

For a better understanding of the scale of both tracking the issue and recalling the potentially impacted vehicles, consider that Toyota’s engine manufacturing plant in Kentucky (their supplier) produces approximately 600,000 engines a year,⁴⁵ and in 2019, Toyota sold over two million vehicles⁴⁶ in the United States. To account for all defective parts, Toyota recalled just over 44,000 vehicles. However, the silver lining to this story is that the final number of vehicles impacted by the defect was minimal: only 250, or about 0.05 percent, of those included in the recall.

This example demonstrates the application of all three of Deming’s principles. First, inspection was built into the manufacturing process by design and occurred before final assembly. And it’s important to note, even with those inspections in place, some defects don’t become apparent until they’ve reached production. Next, the example shows the importance of a strong relationship with suppliers, which made it easier to pinpoint the cause of the engine defect once they appeared after final assembly. In this scenario, the supplier’s adherence to supply chain best practices was as important as the manufacturer’s. Finally, because Toyota tracks and monitors its parts and vehicles, it was possible to use the serial numbers from engines manufactured with the faulty water flow meter and identify only those vehicles with the potential to leak coolant. Toyota then uses all the data from this investigation in its continuous improvement process for manufacturing.

A Recall for Software Manufacturers Already Exists

At first glance, the ability to recall software seems absurd. However, requiring a customer to physically return a software product is too literal an interpretation. Instead, it’s better to consider that software manufacturers share a similar goal to automotive manufacturers: to produce a traceable record of defects and vulnerabilities in their products to reduce costs and respond more quickly, effectively, and efficiently. Thus, the lesson from the previous example for software manufacturers is that a recall process like Toyota’s and other automotive manufacturers’ is already possible. In fact, the best software manufacturers follow a standard process commonly referred to as a Coordinated Vulnerability Disclosure (CVD).⁴⁷

A CVD process is a collaborative approach that typically brings together cybersecurity researchers and software manufacturers to address critical vulnerabilities and provide communication to customers when necessary. To manage the relationship between the members of this group, most software manufacturers publish a vulnerability disclosure policy with criteria and guidance for reporting vulnerabilities, including estimated timelines for remediation and mitigation. At its core, CVD provides a way for software manufacturers to communicate with customers and improve the overall quality and safety of their products. Like the process described in tracing the root cause of Toyota’s defective engines, CVDs are most typically initiated by an external identification of an exploitable vulnerability—a defect. In the case of OSS, this process is already happening and is a recommended best practice in the most widely used projects. To understand how this is already in place today, once again, Log4shell provides a valuable point of analysis.

The identification and announcement of the Log4shell vulnerability was part of a standard CVD process and followed many of the same steps as a product recall. In the case of Log4j, the CVD quickly made the headlines worldwide. Log4j would be hard to miss even for manufacturers not tracking or monitoring their OSS consumption. However, the panic that followed did not stem from the potential severity and exploitability of Log4shell alone. While those aspects were important, an even more fundamental issue was that software manufacturers were unaware of where Log4j was used in their applications or if it was used at all. Unlike Toyota engines, most software manufacturers have no serial number equivalent to connect OSS components like Log4j with impacted products. This gap left software manufacturers only one option: look through every application to find a Log4j dependency, often resorting to scanning the disks of production servers. For organizations with tens of thousands of applications, this is the equivalent of Toyota recalling every vehicle they have ever sold to determine which were affected. Not only would this be unacceptable, but the catastrophic cost would burden Toyota for years.

Defects resulting in a recall, or CVD in the case of software manufacturing, test the strength and security of a supply chain. However, best practices and processes alone are not enough. While Deming’s principles created measurable improvements for automotive manufacturing, another element is at play in the contrasts between Toyota and software manufacturers: responsibility. Software manufacturers must take responsibility for the security of their software from the start. By design,⁴⁸ they must evaluate their suppliers, whether OSS or commercial, and, most importantly, they must continuously track, monitor, and improve their consumption of OSS across all their products and at every stage of the SDLC. Achieving this goal at scale requires a combination of data, processes, best practices, and modern tooling. But most importantly, commitment to the responsibility to deliver safe, secure products.

With these pieces in place, software manufacturers can meet the expectations and standards of their peers in other industries. Software manufacturers will not need to spend months determining which products are affected when the next critical vulnerability, like Log4shell, is identified. Instead, quick and efficient identification will support software manufacturers’ ability to utilize disclosure mechanisms like CVDs and proactively communicate mitigation and remediation steps with their customers. While this is not equivalent to removing products from shelves through physical recalls, better communication can still drive reduced risk for customers in the same manner. Further, with the improved consumption of OSS and attention to the guidance outlined throughout this paper, software manufacturers can work to avoid vulnerabilities in the first place.

Recommendations for Software Manufacturers and Policymakers

Imagine that the next critical OSS vulnerability is identified. Could software manufacturers determine which applications in their portfolio are at risk? Could they determine, based on context, if the vulnerability is exploitable? Could they ensure that future downloads are of the non-vulnerable version? How would (or could) they disclose that information to customers? Based on the available data, the most likely answer is no, or at least not without great difficulty.⁴⁹

Many months have passed since Log4shell, yet teams continue to be affected. As of the writing of this paper, vulnerable versions of Log4j still constitute one-third of all Log4j downloads. The Log4shell vulnerability has been described as “endemic”⁵⁰ and may never go away. Looking beyond Log4j, almost all downloads (96 percent) of vulnerable open source components have a non-vulnerable version available. The logical conclusion is that software manufacturers are unaware of, uninterested in, or perhaps worse, incapable of seriously evaluating their OSS consumption.

The accepted paradigm of inaction and ignorance regarding OSS consumption and software supply chain security is beginning to change. The latest National Cybersecurity Strategy, along with new requirements for government contractors and vendors, is just the first step. Policy and regulations will be revised with even more stringent criteria.⁵¹

Software manufacturers that follow modern supply chain management best practices and principles described in this paper have an opportunity to address liability concerns and protect their customers from risks associated with unmanaged OSS consumption.⁵² Moreover, when critical vulnerabilities occur, software manufacturers can provide effective communication to guide their customers through a recall-like disclosure process that addresses steps for mitigation and remediation.

To drive these improvements, the last section of this paper is separated into two key areas. The first section provides recommendations for software manufacturers to improve their OSS consumption and supply chain security. Aligned with these recommendations, the next section explores potential strategies for future policies and regulations. It is important to note that these recommendations are not a wish list–they are realistic and based on existing best practices utilized in supply chain management across various sectors. Each recommendation represents a reality that can be achieved today through best practices, processes, and tooling.

Software Manufacturer Recommendations

Build security into software products by design.

Customers expect the software products they purchase and use to be secure and safe. The federal government has made it clear that software manufacturers are responsible for ensuring that expectation is met by design. Meeting that expectation means software manufacturers must actively ensure those parts are free from defects. In alignment with the first principle borrowed from Deming’s supply chain management best practices, reducing defects requires attention to open source consumption at the beginning of software development. Waiting until after a product is shipped to identify vulnerabilities is too late. Instead, software manufacturers should create an environment that supports their development teams with the information and context needed to make the best choices when they begin writing code.

When tackling this recommendation, it is important to consider existing developer workflows. If approaches are draconian or overly cumbersome, the loss of developer efficiency can discount the reduction in risk and improved OSS consumption processes. In many cases, developers do not look at OSS consumption in the same way as other forms of procurement. Success requires proper strategies to ensure that changes are not arduous and do not add undue friction for development teams.

Finally, consider that not all vulnerabilities are exploitable in every situation, and as such, some products may ship with OSS vulnerabilities that introduce little to no risk. Regardless, it is important to balance expectations against an organization’s risk tolerance. Every organization will have a different tolerance, but this is not justification to leave tolerance for risk undefined. Software manufacturers must create policies for OSS consumption that match defined risk tolerance and are integrated throughout the SDLC.

Consume only high-value open source software, components, and projects.

The second principle borrowed from Deming focuses on identifying suppliers that produce the best parts and using them exclusively. While “best” can be highly subjective, software manufacturers should prioritize OSS that consistently provides measurable value to the organization and is updated and supported by an active group of contributors. Measuring value starts with identifying known vulnerabilities and includes criteria like update frequency and how long it takes a project or contributor to fix a vulnerability, among others. Once the best option is identified, manufacturers should use it exclusively. For example, utilizing a single logging framework like Log4j across all software products. In doing so, software manufacturers can reduce their overall risk surface and focus on the OSS that best meets their needs.

According to Deming, the relationship with a supplier is as important as the goods produced. Though OSS is not “supplied” in a traditional sense, the principle around relationships stands. Software manufacturers should contribute back to open source projects as much as possible, especially for remediating vulnerabilities. This should be considered an investment in the long-term availability and quality of the product and can reduce downstream risks associated with future vulnerabilities.

Continuously track, monitor, and improve the security of open source software being consumed.

The last principle borrowed from Deming directs software manufacturers to continuously track, monitor, and improve their OSS consumption. While this is not manually feasible for software, many modern tools support the creation of an organization-level manifest. This manifest should include all OSS consumed within the context of specific products and across every stage of the SDLC. Having a list of OSS and where it is used is only the first step. Key criteria such as known vulnerabilities, age, mean remediation time, and other metadata must be tracked to improve choices and aid decision-making.

Finally, software manufacturers must work at an organizational level to limit their exposure to vulnerabilities, which starts with establishing an OSS consumption policy aligned with the organization’s risk tolerance. This policy provides the foundation for broader, organizational-level governance of OSS consumption. The intent is not to create a list of approved components and reprimand teams when an unapproved component is discovered. Instead, OSS consumption policy should guide decision-making for OSS across the SDLC. More importantly, an organizational policy for open source should be used to educate teams and improve OSS consumption in the long term.

Policymaker Recommendations

Hold software manufacturers responsible and accountable via a national standard of care.

In “Tragedy of the Digital Commons” Sharma argues that “Open-source defects should be governed the same way product defects are: when a defect in a product injures a consumer, the law holds every commercial link in the supply chain capable of having identified and remediated the defect accountable.”⁵³ This view represents an expectation of due care by software manufacturers no different than for manufacturers of any physical product. As Sharma points out, software manufacturers have long been able to disclaim liability under outdated interpretations of contract law.⁵⁴ Legal tools like end-user license agreements (EULAs) typically contain indemnity clauses protecting software manufacturers from liability. However, the National Cybersecurity Strategy (NCS) hopes to change this with its call to “hold the stewards of our data accountable … reshape laws that govern liability for data losses and harm caused by cybersecurity errors, software vulnerabilities, and other risks created by software and digital technologies.”⁵⁵

Software manufacturers’ lack of responsibility runs counter to recommendations from Deming’s first principle and CISA’s guidance that products should be secure by design.⁵⁶ While the National Cybersecurity Strategy Implementation Plan’s (NCSIP) Strategic Objective 3.3.1 calls for the development of a software security liability framework and mentions a standard of care, it does not offer substantive details.⁵⁷ To address this conflict, future policy should solidify the recommendations from the NCS and NCSIP by creating a national standard of care that enumerates the responsibility of software manufacturers to:

Identify and evaluate OSS used across their portfolio of products

Catalog collected data for OSS

Define OSS policies and governance standards

Implement continuous vulnerability tracking and monitoring capabilities across the SDLC

Quickly and directly disclose and remediate vulnerabilities

These capabilities would improve security across the board for both OSS and proprietary component software.

Require software manufacturers to demonstrate their approach to vetting OSS used in their products.

Many software manufacturers have no standards for their OSS consumption. The White House Office of Management and Budget (OMB) sets vendor requirements for federal agencies looking to acquire software products based on standards like NIST’s Secure Software Development Framework (SSDF).⁵⁸ To qualify, a vendor must submit a form attesting to the implementation of the required best practices for the software they provide. However, the SSDF has limitations. While the framework provides guidance for software manufacturers to “define security-related criteria for selecting software,” it provides no details as to the potential criteria to be used beyond requiring third parties to attest they meet defined standards. Even under this paradigm, neither the standards nor attestation requirements indicate a software manufacturer’s approach to OSS consumption beyond technical acquisition (repository, download location, etc.).

According to Deming’s second principle, software manufacturers should use the best OSS. The intent of this recommendation is not to define “best.” More important is the process software manufacturers use to evaluate the OSS they consume and how it is measured against their risk tolerance, both of which should be disclosed to customers. A good foundation for these processes can be found in Open Source Security Foundation’s Open Source Consumption Manifesto, which calls upon all software manufacturers to commit to improving their consumption of OSS through fifteen principles and best practices.⁵⁹ With these considerations in mind, future policy should require all software manufacturers to follow expanded standards for OSS software consumption, including evaluation criteria, applied decision-making best practices, and detailed process descriptions. Software procurement and acquisition requirements for vendors and contractors at the federal level should be expanded to include qualified details of a software manufacturer’s organizational OSS consumption policy, including specifics on the criteria, processes, and tools used when consuming OSS. As outlined in the NCSIP’s Strategic Objective 3.5.2, the False Claims Act provides an enforcement mechanism to ensure truthful attestation, holding software manufacturers accountable to these expanded requirements.

Drive software manufacturers to continuously track, monitor, and improve OSS consumption

Strategic Objective 3.3.3 of the National Cybersecurity Strategy Implementation Plan focuses on CVDs.⁶⁰ While imperfect, the current CVD process works well when communicating from upstream (OSS) to downstream (software manufacturers), but only in scenarios where a software manufacturer continuously tracks, monitors, and improves their consumption of OSS. In cases where this is not done and a critical vulnerability, like Log4shell, fails to make headlines, many software manufacturers do not know the potential risk they create for their customers. This is the exact scenario we see based on research demonstrating a significant proportion of OSS is downloaded with known vulnerabilities while non-vulnerable versions are available.

The third principle adapted from Deming recommends an approach to continuously track, monitor, and improve OSS consumption. This will result in a more proactive response, better communication with customers, and closer alignment with the intent of the recall process utilized by automotive manufacturers. To meet this recommendation in the short term, acquisition and procurement policy should require manufacturers to demonstrate CVD processes for responding to and mitigating OSS with known, critical vulnerabilities in their software products. In the longer term, the requirement should evolve to demonstrate alignment with more robust vulnerability reporting and disclosure—for example, the National Institute of Standards and Technology (NIST) Vulnerability Disclosure Report, highlighted in NIST’s Cybersecurity Supply Chain Risk Management Practices for Technology and Management (C-SCRM) and the Secure Software Development Framework,⁶¹ or utilization of the Vulnerability Exploitability eXchange (VEX), currently led by CISA.⁶²

Beyond controls at the federal government level, disclosure and recall processes for software manufacturers should be aligned with a defined standard of care. Combining these approaches provides a more robust mechanism to drive data security and safety standards for software manufacturers in specific industries, such as financial services and the healthcare sector. Recently proposed regulation from the Securities and Exchange Commission (SEC) recommends new cybersecurity risk management and governance standards, including a requirement for public software manufacturers to adopt a more detailed disclosure process.⁶³ In many ways, the SEC has taken this a step further by defining responsibility for public companies and other businesses within their scope of regulation through their proposed requirement that a demonstration of those processes be provided.⁶⁴ Expanding this requirement to adopt disclosure mechanisms specific to OSS vulnerabilities would require software manufacturers to track, monitor, and improve open source consumption in line with the SEC’s more general cybersecurity requirements.

Bringing It All Together

The road to improvement is paved by lessons learned. Change is hard, and defects are a constant threat to delivering safe and secure products. For manufacturers, minimizing defects is attached to the longevity and reputation of their enterprise, which often hinges on avoiding liability as well. In the past, software manufacturers could avoid liability by delivering products without the same standard of care as their manufacturing peers; that option is ending. Today, the US government, along with governments worldwide, has begun implementing policies and regulations to hold manufacturers responsible for safe and secure software. But gaps remain.

The goal is simple: software manufacturers must build security into software products by design, choose the best suppliers, and track and monitor where those parts are used. This squarely places the responsibility for open source consumption and software supply chain security on manufacturers. To address this more holistically, this paper has focused on the importance of OSS consumption as a critical piece to better software supply chain management. The recommendations provided are time-tested approaches deployed in traditional automotive manufacturing. These ideas are not new–they represent what software manufacturers should be doing already. Every recommendation is built on the same principles.

The aim is not to stifle innovation. Instead, it is to unwind an approach that sidesteps responsibility for due care and to encourage proactive, communicative processes. Policy improvements and expanded guidance provide an opportunity to help software manufacturers improve their responses to defects and communication with customers through recall-like capabilities. What is presented in this paper is a win-win. These principles simultaneously help improve open source and proprietary software supply chains while reducing the overall impact and cost of critical vulnerabilities like Log4shell altogether.

Acknowledgments

The authors would like to thank Deborah Bryant, Aeva Black, Maia Hamin, Stewart Scott, Trey Herr, Shane Miller, Jonathan Meadows, Christopher Robinson, Tobie Langel, and several individuals who will remain anonymous for their feedback on earlier versions of this document, as well as individuals who attended a Chatham House rule workshop on the paper. In addition, the authors would like to thank Anais Gonzalez and Donald Partyka for their layout of this issue brief.

About the Authors

Jeff Wayman has spent more than a decade leading digital content and community teams across OSS Security, DevOps, and DevSecOps roles. In his current position, he guides OSS security thought leadership and content strategy for Sonatype. Jeff promotes OSS security awareness through his work with the OpenSSF End Users Working Group and his contributions to the Atlantic Council’s Open Source Policy Network. Jeff is pursuing an MBA at the Gies College of Business at the University of Illinois, Urbana-Champaign, focusing on Digital Marketing and Strategic Innovation.

Brian Fox, Sonatype co-founder and CTO, is a Governing Board member for the Open Source Security Foundation (OpenSSF), a member of the Apache Software Foundation, and former Chair of the Apache Maven project. As a direct contributor to the Maven ecosystem, including the maven-dependency-plugin and maven-enforcer-plugin, he has over twenty years of experience driving the vision behind the project, as well as developing and leading the development of software for organizations ranging from startups to large enterprises. Brian is a frequent speaker at national and regional events including Java User Groups and other development-related conferences.

The Cyber Statecraft Initiative, part of the Atlantic Council Tech Programs, works at the nexus of geopolitics and cybersecurity to craft strategies to help shape the conduct of statecraft and to better inform and secure users of technology.

learn more

Trending Topics