The complex web of software and hardware components and their licensing schemes makes it difficult for healthcare organizations to upgrade or patch systems that prove to be vulnerable.

When your family opened up that brand-new computer when you were a kid, you didn’t think of all of the third-party work that made typing in that first BASIC program possible. There once was a time when we didn’t have to worry about which companies produced all the bits of licensed software or hardware that underpinned our computing experience. But recent malware attacks and other security events have shown just how much we need to care about the supply chain behind the technology we use every day.

The URGENT/11 vulnerability, the subject of a Cybersecurity and Infrastructure Security Agency advisory issued last July, is one of those events. It forces us to care because it affects multiple medical devices. And it serves as a demonstration of how the software component supply chain and availability of support can affect the ability of organizations to update devices to fix security bugs—especially in the embedded computing space.

URGENT/11 is a vulnerability in the Interpeak Networks TCP/IP stack (IPNet), which was licensed out to multiple vendors of embedded operating systems. IPNet also became the main networking stack in Wind River VxWorks, until Wind River acquired Interpeak in 2006 and stopped supporting IPNet. (Wind River itself was acquired by Intel in 2009 and spun off in 2018.) But the end of support didn’t stop several other manufacturers from continuing to use IPNet. When critical bugs were discovered in IPNet, it set off a scare among the numerous medical device manufacturers that run it as part of their product build.

The average medical or Internet of Things (IoT) device relies on multiple free software or open source utilities. These pieces of software are maintained by any number of third parties—often by just one or two people. In the case of Network Time Protocol (ntp)—software that is in billions of devices—its code is maintained by a single person. And when the OpenSSL Heartbleed vulnerability came out in 2014, the OpenSSL project had two developers working on it. While there are many more developers working on it now, the Heartbleed crisis is emblematic of what happens when we use free software in our devices—the software gets adapted, not really patched, and not really maintained on the device, and little benefit goes back to the project.

Patch economics

Companies are under constant pressure to develop products and reduce expenses. To save time to market and reduce costs, hardware manufacturers often build products using reference designs. These designs come with Board Support Packages, which contain the code and drivers needed to successfully install and run an operating system on the given design. Sometimes they also come with utilities to perform diagnostics, hardware debugging, or monitoring on the devices.

But the Board Support Package is not always updated to address vulnerabilities or newer operating systems. This is the case with many Android devices that continue to be used but don’t get software updates—because of kernel changes that the board support packages and drivers do not support. Oftentimes the device manufacturer needs to update these packages for every new version of an operating system. It then needs to rebuild the new version of its operating system and applications on top of it. Third-party components, such as cameras or additional sensors, also need to have their drivers updated. The amount of work needed to do this is significant and requires a degree of testing similar to that of a brand-new device.

Larger manufacturers, such as Samsung, are capable of absorbing the costs and are able to provide device updates at a lower price because they control numerous market segments (display, memory, etc.). Apple is also capable of providing these updates for a number of years because of its control of the supply chain behind its devices, including the processors, and its move away from third-party intellectual property.

But for other manufacturers, the high cost of updating board support packages, associated drivers (when they exist), and applications makes upgrading devices to a whole new version of an operating system difficult. And it often isn’t possible to update even one specific component. As a result, the expectations set by the major software companies don’t carry over well to markets where you don’t sell as many devices, and there is tremendous market pressure to increase earnings.

Medical devices aren’t smartphones

This sort of thing might not be perceived as a huge problem in a consumer device market, where manufacturers try to drive a constant hardware upgrade cycle. But there’s an expectation that medical devices will be used longer than other devices—they’re considered capital expenses, written into construction budgets for new facilities.

Asking medical device vendors to commit to long-term support for components and long-term supply chain support has a corresponding cost that will be borne by end users. Because of the expense of supporting these devices, many organizations will drop manufacturer support and use a third-party company to provide tech support and device management instead. This removes the incentive for manufacturers to provide additional support.

And medical device vendors don’t always have the flexibility to upgrade their underlying platforms because of the way they license components. Since third-party components are usually licensed for a prebuilt function, the license may only allow for the device’s use with a certain version of an operating system or kernel.

While the Linux community has been nothing short of incredible at maintaining older kernel versions and addressing security issues long after newer kernel versions have been released, putting that patched kernel in place takes significant work. There are a lot of dependencies between all the parts, and it’s very difficult to maintain everything to be able to provide security updates for a particular device or operating system as well as Microsoft, Apple, or IBM Red Hat do at scale. And older kernel and library versions mean that newer software isn’t going to be as easy to port over and use, if at all. Getting Apache 2.4 to run on Red Hat Enterprise Linux 5.x, for instance, was an arduous task.