Hacking Exchange from the Outside In

Microsoft Exchange 2019 prior to March 2024 used the Oracle Outside-In libraries to parse specific file types when attached to emails if an attachment inspection mail flow was configured. By default, Exchange has no mail flow rules configured. Several file types were identified to be processed by the server using the Outside-In SDK. Specifically, the following libraries loaded after using the sample Outside-In files as test attachments.

vspdf.dll
vshtml.dll
vseshr.dll
vsw97.dll
vspp12.dll
vsviso.dll
vspp97.dll
vsw12.dll
vsxl5.dll

These libraries are built to parse files and return any plaintext data they can. They live in a folder labelled TE_vXXX in the Exchange installation directory. However, they are repackaged Oracle Outside-In Content Access libraries. You can readily download the Content Access SDK and demo applications from Oracle directly. Other applications integrate this SDK as well, such as Oracle SQL Server and Oracle WebCenter Content Server.

Previous research into Outside-In by Joshua Drake and Will Dorman a decade ago showed that more digging into the framework could be fruitful. Atredis has identified several files that caused the Exchange file scanner to crash when using the OutsideInModule.dll library to parse attachments.

Exchange itself is configured to prefer Outside-In for some filetypes.

-<TypeList Name="PreferOutsideIn" ListType="Allowed">

    <Type Name="Pdf"/>

    <Type Name="Html"/>

</TypeList>

-<TypeList Name="OutsideInOnly" ListType="Allowed">

    <Type Name="AutoCad"/>

    <Type Name="Jpeg"/>

    <Type Name="Tiff"/>

</TypeList>

However, we were unable to reproduce jpg and tiff vulnerabilities we found in the Outside-In libraries through Exchange. We also noticed that jpg and tiff were removed from the file types supported by file inspection by Exchange mail flows sometime after 2021. These configurations relative to tiff and jpg could be old and useless.

In order to fuzz the libraries, we used two different methods. One method was statically instrumenting the Linux version of the Outside-In Content Access libraries, then fuzzing with AFL. The other option chosen was to dynamically instrument and fuzz with Jackalope on Windows.

AFL

In order to fuzz with AFL, we used afl-dyninst. We used an older version of AFL and dyninst in these examples because it's what we've used in the past and knew it worked. However, a coworker has shown that AFL++ has pretty good dyninst support too. In the future, we'll certainly give it a try. To get things running quickly, we will instrument a simple binary shipped with Outside-In as a demo application called memoryio. It simply accepts a file as an argument and spits out any plaintext contents.

DYNINSTAPI_RT_LIB=libdyninstAPI_RT.so afl-dyninst -m 8 -i ../sdk/demo/memoryio -r libvs_viso.so -r libvs_w12.so -r libvs_eshr.so -r libvs_pp12.so -r libvs_pp97.so -o test

Note the -m 8 in the above command. It was found that instrumenting every basic block caused serious instability. Instead, we are instrumenting any basic blocks 8 bytes or larger. During testing, we also noticed that the library will actively write to ~/.oit while running for no useful reason. Creating the directory, but making it read-only, effectively bypassed this. Otherwise it caused stuttering and hangs.

After copying the instrumented libraries into the correct spots, we can run afl-fuzz. We use AFL_SKIP_BIN_CHECK because we used afl-dyninst to instrument an already-compiled utility called memoryio, rather than compiling our own harness.

AFL_SKIP_BIN_CHECK=1 screen afl-fuzz -i in -o out -m none -M mainA:1/3 -- ./test @@ 
AFL_SKIP_BIN_CHECK=1 screen afl-fuzz -i in -o out -m none -M mainB:2/3 -- ./test @@
AFL_SKIP_BIN_CHECK=1 screen afl-fuzz -i in -o out -m none -M mainC:3/3 -- ./test @@
AFL_SKIP_BIN_CHECK=1 screen afl-fuzz -i in -o out -m none -S subX -- ./test @@

We started 3 main fuzzers in parallel, each running on different deterministic phases. In general, running multiple main fuzzers is a bad idea since they will all perform the same work. However, in the configuration we set up, each fuzzer focuses on different deterministic stages. We then set up 10 sub fuzzers. These fuzzers perform a different kind of strategy than the deterministic main fuzzers. They make random changes to the inputs and just see what happens.

The machine we are fuzzing the Linux libraries on has 32 cores, so we have a bit more leg room. After those fuzzers were set up, a slightly different AFL configuration was used for spice.

AFL_SHUFFLE_QUEUE=1 AFL_SKIP_BIN_CHECK=1 screen afl-fuzz -i in -o out -m none -S subXX -- ./test @@

The AFL_SHUFFLE_QUEUE option was added onto 10 more sub fuzzers. This option takes the queue and randomizes the order of the inputs. In general, you shouldn't need to do this. However, we have so many fuzzers running that it's not going to hurt us in this particular instance and could easily help us.

After that, we get about 100 executions per second per fuzzer across 23 fuzzers. It's not amazing, but it'll do. If we wanted to speed things up in the future, we could implement our own harness with AFL persistent mode.

>>> sub4 (0 days, 20 hrs) <<<
cycle 61, lifetime speed 98 execs/sec, path 825/2389 (34%)

Jackalope

Preparing an environment to fuzz with Jackalope was straight forward. This was done by first cloning the most up to date Jackalope repository and following the build instructions. The fuzzing corpus used for most file formats was acquired from strongcourage's fuzzing-corpus repository. Additional corpus for the `xl5` format was also acquired from the internet with google dorks (Ex:?index.of? xls 1999). The batch file used to execute the fuzzer can be seen below:

C:\Users\ali.ahmad\source\repos\Jackalope\build\Release\fuzzer.exe^
    -in IN\pdf ^
    -out Out\pdf^
    -t 5000^
    -nthreads 9^
    -delivery shmem^
    -nargs 2^
    -instrument_module vspdf.dll^
    -target_module memoryio_sharedmem.exe^
    -target_offset 0x2900^
    -dump_coverage^
    -persist^
    -loop^
    -max_sample_size 0x100000^
    -iterations 3000^
    -cmp_coverage^
    instrument_modules_on_load^
    -dict "C:\Users\ali.ahmad\source\repos\AFLplusplus\dictionaries\pdf.dict"^
    -- "D:\test\TE_v.8.5.3.0\memoryio_sharedmem.exe" @@

Shared memory sample delivery was also used improve fuzzer performance as can be seen by the -delivery shmem flag option. The memoryio sample program provided by Oracle as part of its Outside In library was modified to accept shared memory as input. In addition to shared memory delivery, persistent fuzzing was utilized as can be seen by the -persist flag to fuzz in persistent mode for an added performance boost.

Results

We reported to Microsoft three crashes through our vector on Exchange. However, since the issues were in Oracle’s software, Oracle issued the vulnerability ID CVE-2024-21118.

Exchange will not try to use Outside-In on every file type the library itself supports. These crashes were reproduced by sending an email to the Exchange server with the malicious file attached. As stated previously, a mail flow inspection rule must be configured.

Microsoft subsequently disabled the Outside-In libraries in Exchange in the March 2024 Patch Tuesday updates. You can read more about the patches and advisory here. Atredis would specifically like to thank Lisa Olson and the whole Microsoft Security team for their heroic efforts in working through the disclosure.

Be sure to catch Ali talking more in-depth about our bug hunting and debugging process at RVAsec this summer.

vshtml.dll

This crash was a use-after-free.


 # Child-SP          RetAddr               Call Site
00 000000ea`70d5c610 00007ffc`74141366     vshtml!HTMLWToF+0x11773
01 000000ea`70d5c790 00007ffc`74125204     vshtml!HTMLWToF+0x11726
02 000000ea`70d5c7c0 00007ffc`74130bb0     vshtml+0x5204
03 000000ea`70d5cbe0 00007ffc`74139070     vshtml!HTMLWToF+0xf70
04 000000ea`70d5ce30 00007ffc`74148eb5     vshtml!HTMLWToF+0x9430
05 000000ea`70d5cfe0 00007ffc`399db1c2     vshtml!VwStreamRead+0x305
06 000000ea`70d5d180 00007ffc`399d34fb     sccch!CHUnpackageRemoteData+0x7ab2
07 000000ea`70d5d240 00007ffc`399d3048     sccch!CHReadAhead+0xfb
08 000000ea`70d5d2a0 00007ffc`4b634e20     sccch!CHNextItemId+0x158
09 000000ea`70d5d2f0 00007ffc`3cf589dc     sccca!CAReadNext+0x1fe0
0a 000000ea`70d5d730 00007ffc`3cf5a111     OutsideInModule!CreateTextExtractorModule+0x775c
0b 000000ea`70d5e7d0 00007ffc`3d1a9b4a     OutsideInModule!CreateTextExtractorModule+0x8e91
0c 000000ea`70d5e810 00007ffc`3d1a9630     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x7842a
0d 000000ea`70d5e8d0 00007ffc`3d1ab361     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x77f10
0e 000000ea`70d5e990 00007ffc`3d159d47     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x79c41
0f 000000ea`70d5e9d0 00007ffc`3d15e194     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x28627
10 000000ea`70d5ec20 00007ffc`3d15f2db     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2ca74
11 000000ea`70d5ed60 00007ffc`3d15efc2     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2dbbb
12 000000ea`70d5eec0 00007ffc`3d15e775     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2d8a2
13 000000ea`70d5f000 00007ffc`3d13bcbf     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2d055
14 000000ea`70d5f1c0 00007ffc`3d13c8c9     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0xa59f
15 000000ea`70d5f200 00007ffc`3d133757     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0xb1a9
16 000000ea`70d5f3b0 00007ffc`3d133455     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2037
17 000000ea`70d5f4f0 00007ff6`d60a5379     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x1d35
18 000000ea`70d5f5b0 00007ff6`d6092d7f     scanningprocess+0x15379
19 000000ea`70d5f920 00007ffc`936c91ca     scanningprocess+0x2d7f
1a 000000ea`70d5f950 00007ffc`9366bb26     ntdll!TppWorkpExecuteCallback+0x13a
1b 000000ea`70d5f9a0 00007ffc`92c84de0     ntdll!TppWorkerThread+0x686
1c 000000ea`70d5fc90 00007ffc`936dec4b     KERNEL32!BaseThreadInitThunk+0x10
1d 000000ea`70d5fcc0 00000000`00000000     ntdll!RtlUserThreadStart+0x2b

vsxl5.dll

This crash was an invalid write.


 # Child-SP          RetAddr               Call Site
00 0000003a`a65bcb00 00007ff9`e845621e     vsxl5!PutPrintArea+0x2ab8
01 0000003a`a65bcb60 00007ff9`e8455798     vsxl5!PutPrintArea+0x20ce
02 0000003a`a65bcc00 00007ff9`e844bd4e     vsxl5!PutPrintArea+0x1648
03 0000003a`a65bcd70 00007ff9`e845c501     vsxl5+0x1bd4e
04 0000003a`a65bced0 00007ff9`c214af8b     vsxl5!VwStreamSection+0x7e1
05 0000003a`a65bcfa0 00007ff9`c21434fb     sccch!CHUnpackageRemoteData+0x787b
06 0000003a`a65bd060 00007ff9`c21420de     sccch!CHReadAhead+0xfb
07 0000003a`a65bd0c0 00007ff9`d25b2cc4     sccch!CHGetItemId+0x5e
08 0000003a`a65bd100 00007ff9`e9148a07     sccca!CAReadFirst+0xd4
09 0000003a`a65bd200 00007ff9`e914a111     OutsideInModule!CreateTextExtractorModule+0x7787
0a 0000003a`a65be2a0 00007ff9`ba939b4a     OutsideInModule!CreateTextExtractorModule+0x8e91
0b 0000003a`a65be2e0 00007ff9`ba939630     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x7842a
0c 0000003a`a65be3a0 00007ff9`ba93b361     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x77f10
0d 0000003a`a65be460 00007ff9`ba8e9d47     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x79c41
0e 0000003a`a65be4a0 00007ff9`ba8ee194     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x28627
0f 0000003a`a65be6f0 00007ff9`ba8ef2db     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2ca74
10 0000003a`a65be830 00007ff9`ba8eefc2     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2dbbb
11 0000003a`a65be990 00007ff9`ba8ee775     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2d8a2
12 0000003a`a65bead0 00007ff9`ba8cbcbf     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2d055
13 0000003a`a65bec90 00007ff9`ba8cc8c9     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0xa59f
14 0000003a`a65becd0 00007ff9`ba8c3757     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0xb1a9
15 0000003a`a65bee80 00007ff9`ba8c3455     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2037
16 0000003a`a65befc0 00007ff7`12465379     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x1d35
17 0000003a`a65bf080 00007ff7`12452d7f     scanningprocess+0x15379
18 0000003a`a65bf3f0 00007ffa`0fc091ca     scanningprocess+0x2d7f
19 0000003a`a65bf420 00007ffa`0fbabb26     ntdll!TppWorkpExecuteCallback+0x13a
1a 0000003a`a65bf470 00007ffa`0f5c4de0     ntdll!TppWorkerThread+0x686
1b 0000003a`a65bf760 00007ffa`0fc1ec4b     KERNEL32!BaseThreadInitThunk+0x10
1c 0000003a`a65bf790 00000000`00000000     ntdll!RtlUserThreadStart+0x2b

vsPDF.dll

This crash was an invalid read.

 # Child-SP          RetAddr               Call Site
00 000000b2`54e7c070 00007ffc`21288e30     vspdf+0x9a1a
01 000000b2`54e7c160 00007ffc`2127b467     vspdf+0x18e30
02 000000b2`54e7c2b0 00007ffc`2127e596     vspdf+0xb467
03 000000b2`54e7c2e0 00007ffc`21290c3c     vspdf+0xe596
04 000000b2`54e7c8a0 00007ffb`e4f9b1c2     vspdf!PDFSpecialTell+0x5dc
05 000000b2`54e7cdb0 00007ffb`e4f934fb     sccch!CHUnpackageRemoteData+0x7ab2
06 000000b2`54e7ce70 00007ffb`e4f93048     sccch!CHReadAhead+0xfb
07 000000b2`54e7ced0 00007ffb`f7964e20     sccch!CHNextItemId+0x158
08 000000b2`54e7cf20 00007ffb`eb9689dc     sccca!CAReadNext+0x1fe0
09 000000b2`54e7d360 00007ffb`eb96a111     OutsideInModule!CreateTextExtractorModule+0x775c
0a 000000b2`54e7e400 00007ffb`e9ac9b4a     OutsideInModule!CreateTextExtractorModule+0x8e91
0b 000000b2`54e7e440 00007ffb`e9ac9630     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x7842a
0c 000000b2`54e7e500 00007ffb`e9acb361     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x77f10
0d 000000b2`54e7e5c0 00007ffb`e9a79d47     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x79c41
0e 000000b2`54e7e600 00007ffb`e9a7e194     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x28627
0f 000000b2`54e7e850 00007ffb`e9a7f2db     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2ca74
10 000000b2`54e7e990 00007ffb`e9a7efc2     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2dbbb
11 000000b2`54e7eaf0 00007ffb`e9a7e775     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2d8a2
12 000000b2`54e7ec30 00007ffb`e9a5bcbf     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2d055
13 000000b2`54e7edf0 00007ffb`e9a5c8c9     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0xa59f
14 000000b2`54e7ee30 00007ffb`e9a53757     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0xb1a9
15 000000b2`54e7efe0 00007ffb`e9a53455     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x2037
16 000000b2`54e7f120 00007ff6`9fbd5379     Pipeline2!ScanningPipeline::DllCreatePipelineSessionProcessorAdapter+0x1d35
17 000000b2`54e7f1e0 00007ff6`9fbc2d7f     scanningprocess+0x15379
18 000000b2`54e7f550 00007ffc`41a091ca     scanningprocess+0x2d7f
19 000000b2`54e7f580 00007ffc`419abb26     ntdll!TppWorkpExecuteCallback+0x13a
1a 000000b2`54e7f5d0 00007ffc`412b4de0     ntdll!TppWorkerThread+0x686
1b 000000b2`54e7f8c0 00007ffc`41a1ec4b     KERNEL32!BaseThreadInitThunk+0x10
1c 000000b2`54e7f8f0 00000000`00000000     ntdll!RtlUserThreadStart+0x2b