In today’s intricately interconnected and complex software development ecosystem, a single compromised component can trigger a cascade of security breaches across thousands of organizations worldwide. And the cautionary tales keep piling up: In just the past month we’ve witnessed the CrowdStrike incident, where a faulty “channel file,” automatically pushed out to clients, shut down millions of Windows computers, and the “RoguePuppet” vulnerability that an attacker could exploit to add malware to any Puppet Forge module.

The increased complexity and interdependence of software ecosystems makes software supply chain security (SSCS) a critical concern that affects organizations of every size and in every industry. Everyone wants to crowdsource features to save money. Everyone is at risk because of our growing dependency on third-party software, and because of the increasing frequency and sophistication of software supply chain attacks. 

Unfortunately, most organizations aren’t prepared. In a ReversingLabs survey of 321 security and IT professionals, 98 percent said software supply chain issues pose a significant business risk, but just six in 10 said their software supply chain defenses are up to the task of warding off such attacks.

There’s a way forward, but most organizations aren’t walking the path yet. With RL’s new guide, Software Supply Chain Security for Dummies, we lay out that path forward. Here’s why you need to get started — and how to take your first steps toward a modern approach to managing risk from the software you build or buy.

From Crowdstrike to Puppet to 3CX: Three paths to disaster

The security issues surrounding software supply chain security are broad, but most revolve around three issues: Misplaced trust in third-party components, the double-edged sword of automated updates, and challenges in visibility across complex software supply chains.

With the Crowdstrike incident, for example, the law of unintended consequences came into play. The endpoint detection and response company issued a problematic update to a “channel file,” which is part of the behavioral protection mechanisms used by Crowdstrike’s Falcon sensor. It was intended to be a hash update that contained signatures of things the sensor should be looking for. Instead the update, automatically installed on client machines, caused Windows to crash upon bootup, resulting in the infamous “blue screen of death.” 

While most organizations have remote support tools to help resolve problems like this, in this case the update loaded so early the boot process that those remote tools didn’t have a chance to start up. That’s why you heard stories of support technicians at airports having to ascend 10-foot ladders to reach Windows machines mounted behind those arrivals and departures screens. They had to physically touch every machine to fix them. 

Perhaps you were directly affected by this incident. Perhaps not. But either way you should be asking yourself: What are the implications of deep system access for my security software—especially when customers don’t have an option to stop the updates? 

The Puppet Forge vulnerability, on the other hand, fell into the category of a CI/CD pipeline exploitation. It was a GitHub Actions workflow misconfiguration that had the potential for widespread compromise of modules. Much of what’s in Puppet Forge is crowdsourced; it’s a place where people create modules that typically don’t receive a thorough vetting.

Adnan Khan discovered the Puppet Forge vulnerability and wrote about it in his July 2 blog post. He has a tool that he uses to check Git repositories, and when working with it he discovered a flaw in how Puppet set up repositories. By exploiting this flaw he could make changes and pull requests without anyone approving them. 

Armed with this knowledge, he could have poisoned any module. In this case, if you don’t have Puppet pinned to a particular version it will just pull in the module, which could be harvesting data or doing any number of other malicious things. Khan planned to release his tool as open source at Black Hat. Ask yourself: Could my organization have detected this issue?

Perhaps the most stunning incident was the 3CX attack in 2023, a software supply chain compromise that occurred after North Korean attackers gained access to a 3CX employee’s administrator account. From there they gained access by way of a virtual private network (VPN) to the company’s network and compromised the organization’s Windows and macOS build environments. Many companies that used 3CX’s voice over IP (VOIP) software had set the software’s auto-update function on by default, so once the hackers gained control and pushed the modified update out, people just blindly accepted it. The impact was widespread among the company’s more than 600,000 business customers.

We’re told that updates are good, but that’s not always true, as any grizzled Windows administrator can tell you, so you need to change your mindset. You need to ask yourself: Am I allowing auto-updates to software without any vetting or testing on my end? Do I trust that the software vendor is performing testing that meets my standards? 

The Cloudstrike incident was especially pernicious because the architecture didn’t give the users a choice as to whether or not to accept the update. But with 3CX at least customers could have chosen not to turn on auto updates. Those who waited a few days before accepting might have dodged a bullet. 

Want to beat these modern supply chain threats? Change your thinking.

The best way to protect your organization against these types of potential threats is by changing your mindset about software supply chain security. Ask yourself: What am I running in my environment? What am I allowing for updates? Is there something that has low-level control over my systems that could blow up at any moment? What are people or processes putting into my systems, who’s responsible for those updates, and do those entities perform adequate testing before issuing an update?

As a software vendor you have a responsibility to not let your software become a vector for malware. And if you are a software buyer you have a responsibility to do your homework. You can’t just blindly trust vendors and their updates. 

Think about the software you’re using, the modules you’re adding, and the updates your system is accepting. Start thinking holistically about what you’re bringing into your environments, and whether you’ve done the due diligence required to ensure that the software isn’t going to compromise your systems. 

It’s not easy to stand by this mindset. In the past, people who adopted it were often ridiculed for being overly cautious. People made fun of them because they wouldn’t blindly accept updates, but Windows admins don’t do so, and you shouldn’t either. Fortunately, that kind of pushback is fading, but if it does come up, don’t hesitate to regale the criticism with stories of what happened with CrowdStrike, Puppet Forge and 3CX.

For more on how to think through all of these risks, develop a new mindset around software supply chain security, and build an effective SSCS strategy, read our free ebook, Software Supply Chain Security for Dummies, and drill down into Chapter 3: “Managing Third-Party Commercial Software Risk.”

*** This is a Security Bloggers Network syndicated blog from ReversingLabs Blog authored by Joshua Knox. Read the original post at: https://www.reversinglabs.com/blog/are-you-prepared-for-software-supply-chain-threats-its-time-to-think-different