This blog post introduces VMRay’s novel technique for TLS traffic decryption, which is implemented in the hypervisor, without any modifications to the virtual machine. This approach doesn’t modify the traffic like Man-in-the-Middle-based decryption that sandboxes typically use and doesn’t come with the restrictions of API hooking-based solutions.

Background – Why Attackers Transmit Data Over TLS

One of the most important sources of information for malware analysis and detection is network transmissions. Typical malware communication includes downloading malicious payloads, receiving configurations and commands, and exfiltrating stolen user data. In many cases, the data and payload are kept entirely in memory without creating files, so the most feasible way to acquire the information is to extract it from network transmissions.

Malware authors understand the value of the transmitted information and use encrypted communication channels to hide them from the defender. Increasingly, attackers choose to transmit data over TLS. And even when they do not consciously choose to use TLS, they still often use it as a side-effect. This is because attackers very often misuse or exploit legitimate websites to host malicious payloads. At this point, most legitimate websites are served over HTTPS, so by misusing the service, attackers also benefit from TLS, just like the intended visitors of the website.

Defenders need a robust, scalable, and transparent way to inspect the traffic transmitted over TLS.

Traditional Approaches & Drawbacks

Common approaches to inspect the data encrypted using TLS typically involve:

• The Man-in-the-Middle (MitM) approach, which intercepts the traffic between the communication participants and forwards the messages between them.
• Hooking API functions are used to either encrypt or decrypt the data. A hook intercepts the control flow during execution to monitor the data.

Unfortunately, both approaches have their drawbacks.

A MitM approach needs to maintain an encrypted communication channel to both sides, which requires adding an additional certificate to the analysis environment. This has two drawbacks. First, there is an administrative overhead. The user of the system needs to generate and maintain certificates. Second, the generated certificate differs from the actual certificate of the communication participant. Malware can use this difference to detect and evade the analysis environment. For example, malware can validate the certificate and terminate its process if the certificate has expired. Especially certificate pinning, in which properties of valid certificates are further verified, is an obstacle for MitM-based solutions.

The second approach, function hooking, does not involve additional TLS certificates. Instead, hooking-based solutions install hooks that monitor the data sent over the encrypted channel. The hooks redirect the control flow and intercept TLS-related function calls. At this time the raw data is accessible. However, not all functions can be hooked, so the monitoring system might miss important transmissions. Identifying and hooking all the TLS-related functions potentially used by malware is a challenging task that does not scale well. Especially if the malware utilizes third-party libraries for TLS or implements them on its own. If the monitor chooses to hook an upper layer function (such as HttpSendRequest), then the HTTP header is constructed later by the function, making the monitor miss the header. This often contains valuable information, such as the URL or the user agent. In addition, hooking is also susceptible to recognition. Because the installation of hooks and redirections modify the analysis environment malware can detect them.

Solutions based on one of the above approaches fail to meet the requirements to decipher encrypted data while staying fully transparent to the malware and remaining scalable.

Hypervisor-based Decryption – Nonintrusive TLS Visibility

In VMRay Platform v4.2.0, we introduce Nonintrusive TLS Visibility, an approach that solves these challenges. It supports decryption of TLS traffic, regardless of whether the malware uses well-known functions, for example, provided by the native WinAPI, or is equipped with these functions itself. During malware execution, the VMRay Platform records the encrypted network traffic of the virtual machine and extracts the key material by introspecting memory regions from the hypervisor. After execution is finished, it uses the previously extracted key material to decrypt the traffic.

Given the hypervisor-based approach without relying on MitM, this solution does not introduce modifications to the virtual machine. Because there is no MitM, the user no longer has to worry about managing certificates, and the malware can’t detect that the monitoring environment tampered with the connection. It doesn’t come with the drawbacks of hooking-based solutions either: network traffic is monitored even when the malware uses third-party libraries or other rare API functions. The traffic includes all HTTP headers, and monitoring is not exposed to the virtual machine, unlike the agent of a hooking-based sandbox.

The hypervisor-based solution is able to decrypt TLS traffic in a transparent and scalable way.

Nonintrusive TLS Visibility In-Action

Figure 1 shows a typical malicious Microsoft Excel document that uses a macro to deliver a malicious binary through HTTPS. Documents like this are often spread through email and are one of the most common attack vectors.

Because the document uses HTTPS to download the payload, the transmitted data is encrypted. Eavesdropping the communication would be enough to provide limited meta-information, such as the connected domain and the size of the downloaded data, but it misses crucial details. Thanks to the TLS decryption, we can see the exact URL, and extract the downloaded payload (a malicious DLL) for further analysis (Figure 2). VMRay Platform can use the extracted data for detection, and the VMRay Analyzer report displays it on the Network tab.

In addition to showing the decrypted traffic in the Network tab, VMRay Analyzer provides an  file the user can download. Based on the extracted key material, it generates a key log file containing shared secrets which are then embedded automatically into the PCAP file. Tools like Wireshark that support TLS decryption using an embedded key log file, can then decrypt the traffic (Figure 3). This allows users to further examine the traffic if desired.

Experience our nonintrusive TLS Visibility first hand by requesting a trial of VMRay. Get started today!