Update Monday, July 24, 2023: 

After originally publishing an analysis of unpatched servers on Fri 21 Jul and giving administrators the weekend to patch, we're updating this post to note the vulnerable route, /gwtest/formssso. The research team at Assetnote has published their analysis confirming the same path.

Published Friday, July 21, 2023:

Bishop Fox developed an exploit for CVE-2023-3519, a stack overflow in Citrix ADC Gateway that allows remote code execution. There are 61,000 affected appliances exposed on the internet, and roughly 53% of them are currently unpatched. You should patch yours now.

A side note on what to call this tech stack: Is it Citrix or NetScaler? ADC or Gateway? Is Citrix ADC configured with the Gateway—or is it AAA? Oh, so NetScaler is the old name…but wait, NetScaler is back? For simplicity, we’ll broadly refer to this appliance as Citrix ADC, and to the specific vulnerable interface as Citrix Gateway.

The Exploit

FIGURE 1 – Remote code execution via CVE-2023-3519 on Citrix ADC VPX version 13.1-48.47

The vulnerability is a simple unauthenticated stack overflow. This is made significantly worse by the fact that exploit mitigations do not protect the vulnerable function on some versions. The vulnerable binary is compiled without PIE and with an executable stack, and on the VPX version, there is no stack canary. As a result, exploitation is trivial. Our exploit cleanly returns without crashing the vulnerable process.

The vulnerability that we identified is different from the one identified by Rapid7 in this AttackerKB article and by AssetNote in their analysis, which required SAML to be enabled. The vulnerability we identified only requires the device to be configured as a Gateway or AAA virtual server, and to expose a specific vulnerable route that seems to be enabled by default on some installations, but not others (we’re not yet sure what causes this variance). Given the lack of SAML requirement, we believe that this stack overflow is CVE-2023-3519, and the SAML parser bug is a separate vulnerability which was silently patched without an associated advisory.

At this time, we’re not going to disclose the aforementioned vulnerable route—but we’ve shared this information with our friends at GreyNoise Intelligence who’ve added a tag to help defenders track exploitation attempts. Thanks especially to Remy from GreyNoise for collaborating with us during the initial identification and analysis of this vulnerability.

Searching Shodan

Locating this tech stack on Shodan is pretty straightforward. Searching for the most common titles of Citrix Gateway login pages, about 61,000 devices are found.

for TITLE in 'NetScaler AAA' 'Citrix Gateway' 'NetScaler Gateway'; do
    echo -n "$TITLE:"
    shodan count "title:\"$TITLE\""
done |
    sort -t : -nrk2 |
    awk -F : '{SUM += $NF; print $0} END {print "Total:" SUM}' |
    column -ts:

Citrix Gateway     38544
NetScaler AAA      19551
NetScaler Gateway  3065
Total              61160

Note that others have searched for this application using JARM hashes, favicon hashes, etc.—but searching title seems to do the trick and comes back with similar results.

Finding Unpatched Devices

By inspecting Citrix’s released software images, we know that patched ADC releases were packaged in July 2023. If we search Shodan for that month in the Last-Modified HTTP response header, we can find devices that have been patched.

Our analysis shows 53% (32k) of internet-exposed Citrix ADC appliances to be unpatched, and a smaller subset of 35% (21k) to be unpatched and exposing the vulnerable route.

Below, we can see these installations graphed by their Last-Modified header values:

FIGURE 2 - Citrix ADC installations from January 2015 to July 2023

At a glance, this doesn’t look too bad. We see a huge spike of installations clustered around summer 2023, which would imply that many have been patched. To see the outliers installed prior to 2023, let’s take a logarithmic view of this data:

FIGURE 3 - Logarithmic view of Citrix ADC installations from January 2015 to July 2023

Just like we did in our recent FortiOS analysis, let’s take a sample of Citrix ADC devices and graph them in a stacked bar plot according to their major versions. This time, we’ll skip the linear view and head straight for the logarithmic view.

FIGURE 4 – Logarithmic view of Citrix ADC installations of versions 10–13 from January 2015 to July 2023

Yikes. An unpatched security appliance from way back in 2015? Network administrators need to take timely patching seriously—let alone leaving an appliance in neglect for eight years. You'll probably find more shells in there than a walk on the beach.

Conclusion

At Bishop Fox, we want to see our customers keep their most important assets patched in a timely manner, especially those with vulnerabilities that are proven to be exploitable. If you’ve got a Citrix ADC installation, please follow Citrix’s advisory for this issue and upgrade your firmware immediately. Happy patching!