Authored by Yun Zheng Hu

Recently, two critical vulnerabilities were reported in Citrix ADC and Citrix Gateway; where one of them was being exploited in the wild by a threat actor. Due to these vulnerabilities being exploitable remotely and given the situation of past Citrix vulnerabilities, RIFT started to research on how to identify the exact version of Citrix ADC and Gateway servers on the internet so that we could inform customers if they hadn’t patched yet.

The exact version information is helpful to determine whether a server is still vulnerable to particular vulnerabilities. We also used version information to derive version statistics and version adoption over time.

In the first part of this blog, we go into how we acquired and analysed Citrix ADC disk images for version identification. Then we go into how we found a way to determine the version build date and how we used the build dates to download missing Citrix ADC images to aid our version identification research. The last part goes into the version statistics of Citrix ADC servers on the internet and how we used this to measure the version adoption on the internet.

Skip to the end if you are interested in the statistics rather than the technical details of Citrix ADC version identification.

CVE-2022-27510 – Unauthorized access to Gateway user capabilities

On November 8th 2022, Citrix published a security bulletin for CVE-2022-27510, a critical authentication bypass vulnerability affecting Citrix ADC (formerly known as NetScaler) and Citrix Gateway. For this to be exploitable, the server must be configured as a Gateway (SSL VPN, ICA Proxy, CVPN, RDP Proxy).

CVE-2022-27518 – Unauthenticated remote arbitrary code execution

Less than a month later, on December 13th 2022, the National Security Agency (NSA) released a Cybersecurity Advisory that APT5 is actively exploiting Citrix ADC servers. However this advisory does not mention a specific CVE that is being abused but most likely a new vulnerability as on the same day Citrix published a blog with guidance and a new security bulletin detailing CVE-2022-27518, which is a new vulnerability and not to be confused with CVE-2022-27510. For this to be exploitable, the Citrix ADC or Gateway server must be configured as a SAML Service Provider or SAML Identity Provider.

Finding Citrix ADC & Gateway Servers on the Internet

Citrix ADC and Gateway servers are commonly internet-facing due to the nature of the appliance. For example, services like Shodan and Censys regularly scan the internet and identify these servers. Using this information, we can build a list of Citrix ADC & Citrix Gateway servers with the SSL VPN / Gateway service exposed to the internet. We used this to create an initial list of servers, and we found around 28.000 servers on the internet as of November 11th 2022.

Version identification

Sadly, the exact version information is not available in the HTTP response of a Citrix ADC or Gateway server. However, we noticed that there is an MD5 hash-like value in the HTTP body when requesting the /vpn/index.html URL:

Here we see the parameter ?v=6e7b2de88609868eeda0b1baf1d34a7e appended to several resource URLs. We extracted these hashes from Censys scan data to create a list of the most common version hashes. We found around 100 unique version hashes.

To see if we can map the version hash to exact versions, we begin by first spinning up our own Citrix ADC server to start exploring wether this version hash in this HTML page is static or generated.

Cloud Marketplace

More server appliances become available in the cloud, and Citrix ADC is no exception. You can easily find it in your favourite cloud marketplace. So gone are the days of the time-consuming process of downloading images and spinning up a VM to install the application; we can do this directly in the cloud! We used the Google Cloud Marketplace, but it’s also available on AWS and Azure.

After deploying Citrix ADC with a single click from the Cloud Marketplace, we log in to the shell and explore the file system to see if we can find the index.html page and if it contains a hash value.

SSH into the VM and show the version, looks like we are on version 13.1 build 33.47:

[email protected]:~ (rift-citrix-362712)$ gcloud ssh citrix-adc-vpx-instance
###############################################################################
#                                                                             #
#        WARNING: Access to this system is for authorized users only          #
#         Disconnect IMMEDIATELY if you are not an authorized user!           #
#                                                                             #
###############################################################################

 Done
> show version
        NetScaler NS13.1: Build 33.47.nc, Date: Sep 23 2022, 13:12:49   (64-bit)
 Done

Type shell to enter the shell, and let’s find all index.html files:

> shell
[email protected]# uname -a
FreeBSD ns 11.4-NETSCALER-13.1 FreeBSD 11.4-NETSCALER-13.1 #0 e5f9d90507ab(heads/artesa_33_47)-dirty: Fri Sep 23 13:13:05 PDT 2022     [email protected]:/usr/obj/usr/home/build/adc/usr.src/sys/NS64  amd64

[email protected]# find / -name 'index.html'
/netscaler/ns_gui/admin_ui/gui_v2/swagger_ui/index.html
/netscaler/ns_gui/vpn/index.html
/var/netscaler/gui/vpn/index.html
/var/netscaler/gui/admin_ui/gui_v2/swagger_ui/index.html
/var/netscaler/logon/LogonPoint/index.html
/var/python/lib/python3.7/site-packages/djangorestframework-3.11.0-py3.7.egg/rest_framework/templates/rest_framework/docs/index.html
/var/python/lib/python3.7/site-packages/Django-3.0.5-py3.7.egg/django/contrib/admin/templates/admin/index.html
/var/python/lib/python3.7/site-packages/Django-3.0.5-py3.7.egg/django/contrib/admindocs/templates/admin_doc/index.html

Looks like there are several directories with index.html, let’s check /vpn/index.html:

[email protected]# head /netscaler/ns_gui/vpn/index.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XDEV_HTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Citrix Gateway</title>
<link rel="SHORTCUT ICON" href="/vpn/images/AccessGateway.ico" type="image/vnd.microsoft.icon">
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<META content=noindex,nofollow,noarchive name=robots>
<link href="/vpn/js/rdx/core/css/rdx.css?v=7afe87a42140b566a2115d1e232fdc07" rel="stylesheet" type="text/css"/>
<link href="/logon/themes/Default/css/base.css?v=7afe87a42140b566a2115d1e232fdc07" rel="stylesheet" type="text/css" media="screen" />

Bingo! The index.html file contains the hash 7afe87a42140b566a2115d1e232fdc07, and if we search for this value on Censys, we get multiple results which is promising. So we can assume that version 13.1-33.47 maps to this hash value.

Let’s check which other files contain this hash value for good measure:

[email protected]# grep -r 7afe87a42140b566a2115d1e232fdc07 /var/netscaler | cut -d: -f1 | sort -u
/var/netscaler/gui/epa/epa.html
/var/netscaler/gui/epa/errorpage.html
/var/netscaler/gui/epa/posterrorpage.html
/var/netscaler/gui/vpn/index.html
/var/netscaler/gui/vpn/loading.html
/var/netscaler/gui/vpn/logout.html
/var/netscaler/gui/vpn/tmindex.html
/var/netscaler/gui/vpn/tmlogout.html
/var/netscaler/gui/vpns/choices.html
/var/netscaler/gui/vpns/f_ndisagent.html
/var/netscaler/gui/vpns/f_services1.html
/var/netscaler/gui/vpns/f_services_linux.html
/var/netscaler/gui/vpns/j_services.html
/var/netscaler/gui/vpns/m_services.html
/var/netscaler/gui/vpns/navui/refresh.html
/var/netscaler/gui/vpns/nohomepage.html
/var/netscaler/gui/vpns/postepa.html

When installing the application from the Google Cloud Marketplace, it does not give the option to choose a version to install. But of course, we want to install other versions to start building a list of known version hashes.

We noticed that a deployment.zip file can be downloaded from the Google Cloud Marketplace:

After unpacking deployment.zip we see that this contains Terraform scripts to deploy the cloud application and references the disk image it uses for installation. Luckily, Citrix also left a list in a Jinja template of other disk image names referencing different Citrix ADC versions.

Can these disk images be downloaded? Yes, it turns out you can!

Acquiring Cloud images

We used the following gcloud commands to download Citrix ADC disk images:

• gcloud compute images create, to download the image in our own Google Cloud project. This allows you to choose this image when you create a new VM. However, this does not make the image directly accessible for reading using other tools, for that we need to export it first.
• gcloud compute images export, this exports the given image to a Google Cloud Storage (GCS) bucket using a different format such as qcow2 or vmdk.

The raw disk image is 20 GB but exporting it as qcow2 format we can reduce this to only 2 GB.

The following shell script does both steps in one script:

#!/bin/sh
#
# Export a Citrix ADC image to a Google Cloud Storage bucket
#
# Usage:
#   ./export-citrix-image.sh <image name> <bucket_path>
# Example:
#   ./export-citrix-image.sh citrix-adc-vpx-10-standard-13-0-85-19 gs://my-bucket
image="$1"
gcs_bucket="$2"

gcloud compute images create "$image" --source-image https://www.googleapis.com/compute/v1/projects/citrix-master-project/global/images/$image --verbosity debug
gcloud compute images export --destination-uri "$gcs_bucket/$image.qcow2"  --image "$image" --export-format qcow2

Now that we exported some known images from deployment.zip to a GCS bucket, we can process them in bulk. Of course, having them archived and preserved can also be helpful for future research, especially for vulnerable versions, as they tend to get deleted.

This technique can also be used for other appliances in the Google Cloud Marketplace.

Dissect to the rescue!

What better way to process disk images in bulk than dogfooding our own opensource dissect framework. While Citrix ADC runs on FreeBSD, dissect can read its UFS/FFS file systems just fine. After we fixed a few bugs that is.

We can also do everything in a Cloud Shell, without spinning up an extra VM.

Let’s mount our GCS bucket containing the qcow2 images first using gcsfuse:

[email protected]:~ (rift-citrix-362712)$ mkdir bucket

[email protected]:~ (rift-citrix-362712)$ gcsfuse my-citrix-adc-bucket bucket
2022/12/11 15:48:25.442402 Start gcsfuse/0.41.9 (Go version go1.18.4) for app "" using mount point: /home/yun/bucket
2022/12/11 15:48:25.462744 Opening GCS connection...
2022/12/11 15:48:25.557883 Mounting file system "my-citrix-adc-bucket"...
2022/12/11 15:48:25.563799 File system has been successfully mounted.

[email protected]:~ (rift-citrix-362712)$ ls -1 bucket/citrix*.qcow2
citrix-adc-vpx-10-standard-13-0-83-27.qcow2
citrix-adc-vpx-10-standard-13-0-87-9.qcow2
citrix-adc-vpx-10-standard-13-0-88-14.qcow2
citrix-adc-vpx-10-standard-13-1-21-50.qcow2
citrix-adc-vpx-10-standard-13-1-33-52.qcow2
citrix-adc-vpx-byol-13-0-76-31.qcow2
citrix-adc-vpx-byol-13-0-79-64.qcow2
citrix-adc-vpx-byol-13-0-82-45.qcow2
citrix-adc-vpx-byol-13-0-88-16.qcow2
citrix-adc-vpx-byol-13-1-33-49.qcow2
citrix-adc-vpx-byol-13-1-33-52.qcow2
citrix-adc-vpx-byol-13-1-33-54.qcow2
citrix-adc-vpx-byol-13-1-37-38.qcow2
citrix-adc-vpx-express-13-0-83-29.qcow2
...

Now we install the latest version of dissect using pip3 in a virtualenv:

[email protected]:~ (rift-citrix-362712)$ python3 -mvenv dissect

[email protected]:~ (rift-citrix-362712)$ source dissect/bin/activate
(dissect) [email protected]:~ (rift-citrix-362712)$ pip3 install --pre dissect

Dissect will install different command line tools. One of these tools is called target-shell, which can read disk images and file systems in various formats and gives you a shell-like interface to explore the file system.

Now let’s open a disk image using target-shell, we specify the -q flag to hide some warnings:

Dissect does not (yet!) support Citrix ADC, so instead it gives us access to the discovered filesystems. We see two partitions, and we found the first one is the /boot partition and second one is the /var partition. target-shell has a basic find command, but the output can be piped to an external tool such as grep:

citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /> cd fs1

citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> find . | grep index.html
/fs1/netscaler/gui/vpn/index.html
/fs1/netscaler/gui/vpn/tmindex.html
/fs1/netscaler/gui/admin_ui/gui_v2/swagger_ui/index.html
^C

Let’s cat the first index.html file (grep is used to limit the output for this example):

citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> cat /fs1/netscaler/gui/vpn/index.html | grep ?v= | head -n1
<link href="/vpn/js/rdx/core/css/rdx.css?v=c9e95a96410b8f8d4bde6fa31278900f" rel="stylesheet" type="text/css"/>
citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1>

Nice, we find that for version 13.0-83-27 the version hash is c9e95a96410b8f8d4bde6fa31278900f.

Not everyone realizes this, but with just this one command, we loaded a FreeBSD image in qcow2 disk format containing multiple UFS/FFS2 partitions by using Python and dissect in a Cloud Shell. No hassle of installing additional tools and performing tedious steps to mount images. The future is now!

To even further automate this we can use the dissect Python API or we can make a simple shell oneliner using target-fs, which can execute some basic commands against a disk image:

It’s good to mention that at this point, we also started to investigate if the version hash can be calculated by MD5 summing variants of the version string, but without any luck.

Identifying the build date

After processing our acquired cloud images, we found that we still had version hashes from the internet without a known version, so it seemed that not all versions were listed or available as a cloud image. We did manage to acquire some cloud images by guessing the image name, but this was not enough to fill in the gaps.

We finally resorted to the Citrix downloads page to look for other versions we didn’t have yet. A Citrix account is required but it’s open for registration. However, it seemed that not every version that was released is listed on the Citrix downloads page, especially older builds that were replaced by a new build. We found a GitHub project that scraped Citrix download links that were useful for our research, and we found that these links are still valid (after logging in).

While our known version list kept growing, we still missed certain hashes that were quite common in the dataset. We naturally wanted to know which version that was, and at this point, we found an interesting way to determine the approximate build date of the Citrix ADC server version.

In the disk image we found the gzip-compressed file named rdx_en.json.gz under the vpn subdirectory:

citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> find . | grep rdx_en.json.gz
/fs1/netscaler/gui/vpn/js/rdx/core/lang/rdx_en.json.gz
/fs1/netscaler/gui/admin_ui/rdx/core/lang/rdx_en.json.gz
^C

citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /> ls -la /fs1/netscaler/gui/vpn/js/rdx/core/lang | grep rdx_en
-rw-r--r-- 1001  513     35 2021-09-27T14:01:20 rdx_en.json.gz

Let’s run the file command on this gzip file:

citrix-adc-vpx-10-standard-13-0-83-27.qcow2 /fs1> file /fs1/netscaler/gui/vpn/js/rdx/core/lang/rdx_en.json.gz
/fs1/netscaler/gui/vpn/js/rdx/core/lang/rdx_en.json.gz: gzip compressed data, was "rdx_en.json", last modified: Mon Sep 27 14:01:20 2021, from Unix

Last modified Mon Sep 27 14:01:20 2021. Cool, we retrieve the timestamp of when this gzip file was created and we found that this timestamp is an accurate depiction of when the version was created/released, pretty neat! This JSON file seems to be used for translations purposes, but in newer versions this file is just an empty gzipped JSON dictionary.

Because this rdx_en.json.gz file resides in the vpn sub-directory, it can also be downloaded remotely by accessing the following URI /vpn/js/rdx/core/lang/rdx_en.json.gz

We now have a list of version hashes, known versions, and approximate build dates. Using the build dates we can now deduce the approximate version of a version hash that we don’t know the version of yet.

For example, the version hash 4434db1ec24dd90750ea176f8eab213c was still missing its version number, and we already processed all the cloud images and available download links from the Citrix download page. But armed with the knowledge of the build date 2022-06-29 13:46:08 of this version hash, we can deduce that this build date falls between the build dates of the known versions 12.1-65.15 and 12.1-65.25. Table for clarity:

build date	version hash	version
2022-05-22 19:18:31	fbdc5fbaed59f858aad0a870ac4a779c	12.1-65.15
2022-06-29 13:46:08	4434db1ec24dd90750ea176f8eab213c	??
2022-10-04 16:11:03	f063b04477adc652c6dd502ac0c39a75	12.1-65.25

Ok, the possible versions can be: 12.1-65.16 to 12.1-65.24.

These versions are not listed on the download page but does a version within this range exist and can it still be downloaded? By looking at the Citrix download links of known versions we see that the Download ID looks to be incremental, and that the files have a specific format. For example:

URL format example: https://downloads.citrix.com/[DOWNLOAD_ID]/build-[VERSION]_nc_64.tgz

https://downloads.citrix.com/20651/build-12.1-65.15_nc_64.tgz <-- known url
https://downloads.citrix.com/20???/build-12.1-65.??_nc_64.tgz <-- enumerate this url
https://downloads.citrix.com/21408/build-12.1-65.25_nc_64.tgz <-- known url

Can we enumerate the URL and then download the file? The answer is yes, by using a small Python script to enumerate the download link and version, we find that the following URL returns a 200 OK:

https://downloads.citrix.com/20929/build-12.1-65.17_nc_64.tgz

After downloading build-12.1-65.17_nc_64.tgz, we confirmed that version 12.1-65.17 maps to the version hash 4434db1ec24dd90750ea176f8eab213c. This technique proved to be very useful for filling in most of the gaps in versions in our dataset, with a few missing.

Our compiled list of version hashes, approximate build dates, and versions can be found in this gist: https://gist.github.com/fox-srt/c7eb3cbc6b4bf9bb5a874fa208277e86

Version statistics

Now that we have mapped most of the known version hashes to a version, we can measure how many versions there are active on the internet, and if they are still vulnerable to CVE-2022-27510 or CVE-2022-27518.

The following graph shows the Top 20 active versions on the internet, and also shows if that version is vulnerable to any of the two recent CVEs:

We see that the majority is on version 13.0-88.14, which is not vulnerable to either of the two CVEs. The runner up is version 12.1-65.21 which is not vulnerable to CVE-2022-27510, but it is to CVE-2022-27518. There are also many servers that do not return a version hash at all so for those servers we cannot identify the exact version.

Note that for CVE-2022-28518 a pre-condition of SAML is required for it to be exploitable, so knowing only the version does not fully indicate if the server can be exploited but it’s still a good indicator that it should be updated.

The following graph shows the Top 9 countries using Citrix ADC / Gateway and how many servers are still vulnerable to the two critical CVEs. For most countries we see an obvious drop of servers vulnerable to CVE-2022-27518 after the NSA and new Citrix advisory publication.

This graph shows the Top 20 countries using Citrix ADC / Gateway and how many servers are properly updated so they are protected against both CVEs.

Conclusion

In this blog, we’ve shown how we performed the version identification of Citrix ADC and Citrix Gateway servers by analysing disk images exported from Google Cloud Marketplace using dissect. We also demonstrated that gzip files can be helpful for timestamp information and how we utilised this to find and download missing Citrix ADC builds.

Finally, we used the version identification data to measure the versions of internet-facing Citrix ADC and Gateway servers over time and see that the NSA and Citrix advisory really helped with updates. However, some servers remain vulnerable to CVE-2022-27510 or CVE-2022-27518.

We hope this blog creates extra awareness for these two Citrix CVEs and that our research on version identification contributes to future studies.