Sysmon (System Monitor) is a well-known and widely used Windows logging utility providing valuable visibility into core OS (operating system) events. From a defender’s perspective, the presence of Sysmon in an environment greatly enhances detection and forensic capabilities by logging events involving processes, files, registry, network connections and more.
Since Sysmon 11 (released April 2020), the FileDelete
event provides the capability to retain (archive) deleted files, a feature we especially adore during active compromises when actors drop-use-delete tools. However, as duly noted in Sysmon’s documentation, the usage of the archiving feature might grow the archive directory to unreasonable sizes (hundreds of GB); something most environments cannot afford.
This blog post will cover how, through a Windows-native feature (WMI event consumption), the Sysmon archive can be kept at a reasonable size. In a hurry? Go straight to the proof of concept!
Typical Sysmon deployments require repeated fine-tuning to ensure optimized performance. When responding to hands-on-keyboard attackers, this time-consuming process is commonly replaced by relying on robust base-lined configurations (some of which open-source such as SwiftOnSecurity/sysmon-config or olafhartong/sysmon-modular). While most misconfigured events have at worst an impact on CPU and log storage, the Sysmon file archiving can grind a system to a halt by exhausting all available storage. So how could one still perform file archiving without risking an outage?
While searching for a solution, we defined some acceptance requirements. Ideally, the solution should…
A common proposed solution would be to rely on a scheduled task to perform some clean-up activities. While being Windows-native, this execution method is “dumb” (schedule-based) and would execute even without files being archived.
So how about WMI event consumption?
WMI (Windows Management Instrumentation) is a Windows-native component providing capabilities surrounding the OS’ management data and operations. You can for example use it to read and write configuration settings related to Windows, or monitor operations such as process and file creations.
Within the WMI architecture lays the permanent event consumer.
You may want to write an application that can react to events at any time. For example, an administrator may want to receive an email message when specific performance measures decline on network servers. In this case, your application should run at all times. However, running an application continuously is not an efficient use of system resources. Instead, WMI allows you to create a permanent event consumer. […]
A permanent event consumer receives events until its registration is explicitly canceled.
docs.microsoft.com
Leveraging a permanent event consumer to monitor for file events within the Sysmon archive folder would provide optimized event-based execution as opposed to the scheduled task approach.
In the following sections we will start by creating a WMI event filter intended to select events of interest; after which we will cover the WMI logical consumer whose role will be to clean up the Sysmon archive.
A WMI event filter is an __EventFilter
instance containing a WQL (WMI Query Language, SQL for WMI) statement whose role is to filter event tables for the desired events. In our case, we want to be notified when files are being created in the Sysmon archive folder.
Whenever files are created, a CIM_DataFile
intrinsic event is fired within the __InstanceCreationEvent
class. The following WQL statement would filter for such events within the default C:\Sysmon\
archive folder:
SELECT * FROM __InstanceCreationEvent WHERE TargetInstance ISA 'CIM_DataFile' AND TargetInstance.Drive='C:' AND TargetInstance.Path='\\Sysmon\\'
Intrinsic events are polled at specific intervals. As we wish to ensure the polling period is not too long, a WITHIN
clause can be used to define the maximum amount of seconds that can pass before the notification of the event must be delivered.
The beneath query requires matching event notifications to be delivered within 10 seconds.
SELECT * FROM __InstanceCreationEvent WITHIN 10 WHERE TargetInstance ISA 'CIM_DataFile' AND TargetInstance.Drive='C:' AND TargetInstance.Path='\\Sysmon\\'
While the above WQL statement is functional, it is not yet optimized. As an example, if Sysmon came to archive 1000 files, the event notification would fire 1000 times, later resulting in our clean-up logic to be executed 1000 times as well.
To cope with this property, a GROUP
clause can be used to combine events into a single notification. Furthermore, to ensure the grouping occurs within timely manner, another WITHIN
clause can be leveraged. The following WQL statement waits for up to 10 seconds to deliver a single notification should any files have been created in Sysmon’s archive folder.
SELECT * FROM __InstanceCreationEvent WITHIN 10 WHERE TargetInstance ISA 'CIM_DataFile' AND TargetInstance.Drive='C:' AND TargetInstance.Path='\\Sysmon\\' GROUP WITHIN 10
To create a WMI event filter we can rely on PowerShell’s New-CimInstance
cmdlet as shown in the following snippet.
$Archive = "C:\\Sysmon\\" $Delay = 10 $Filter = New-CimInstance -Namespace root/subscription -ClassName __EventFilter -Property @{ Name = 'SysmonArchiveWatcher'; EventNameSpace = 'root\cimv2'; QueryLanguage = "WQL"; Query = "SELECT * FROM __InstanceCreationEvent WITHIN $Delay WHERE TargetInstance ISA 'CIM_DataFile' AND TargetInstance.Drive='$(Split-Path -Path $Archive -Qualifier)' AND TargetInstance.Path='$(Split-Path -Path $Archive -NoQualifier)' GROUP WITHIN $Delay" }
The WMI logical consumer will consume WMI events and undertake actions for each occurrence. Multiple logical consumer classes exist providing different behaviors whenever events are received, such as:
ActiveScriptEventConsumer
which executes a predefined VBS script.LogFileEventConsumer
which writes a customized message to a text log file.NTEventLogEventConsumer
which logs a specific message in the Windows event log.SMTPEventConsumer
which sends an email using SMTP.CommandLineEventConsumer
which launches an arbitrary process.The last CommandLineEventConsumer
class is particularly interesting as it would allow us to run a PowerShell script whenever files are archived by Sysmon (a feature attackers do enjoy as well).
The first step on our PowerShell code would be to obtain a full list of archived files ordered from oldest to most recent. This list will play two roles:
While getting a list of files is easy through the Get-ChildItem
cmdlet, sorting these files from oldest to most recently archived requires some thinking. Where common folders could rely on the file’s CreationTimeUtc
property, Sysmon archiving copies this file property over. As a consequence the CreationTimeUtc
field is not representative of when a file was archived and relying on it could result in files being incorrectly seen as the oldest archives, causing their premature removal.
Instead of relying on CreationTimeUtc
, the alternate LastAccessTimeUtc
property provides a more accurate representation of when a file was archived. The following snippet will get all files within the Sysmon archive and order them in a FIFO-like fashion.
$Archived = Get-ChildItem -Path 'C:\\Sysmon\\' -File | Sort-Object -Property LastAccessTimeUtc
Once the archived files listed, the folder size can be computed through the Measure-Object
cmdlet.
$Size = ($Archived | Measure-Object -Sum -Property Length).Sum
All that remains to do is then loop the archived files and remove them while the folder exceeds our desired quota.
for($Index = 0; ($Index -lt $Archived.Count) -and ($Size -gt 5GB); $Index++) { $Archived[$Index] | Remove-Item -Force $Size -= $Archived[$Index].Length }
In some situations, Sysmon archives a file by referencing the file’s content from a new path, a process known as hard-linking.
A hard link is the file system representation of a file by which more than one path references a single file in the same volume.
docs.microsoft.com
As an example, the following snippet creates an additional path (hard link) for an executable. Both paths will now point to the same on-disk file content. If one path gets deleted, Sysmon will reference the deleted file by adding a path, resulting in the file’s content having two paths, one of which within the Sysmon archive.
:: Create a hard link for an executable. C:\>mklink /H C:\Users\Public\NVISO.exe C:\Users\NVISO\Downloads\NVISO.exe Hardlink created for C:\Users\Public\NVISO.exe <<===>> C:\Users\NVISO\Downloads\NVISO.exe :: Delete one of the hard links causing Sysmon to archive the file. C:\>del C:\Users\NVISO\Downloads\NVISO.exe :: The archived file now has two paths, one of which within the Sysmon archive. C:\>fsutil hardlink list Sysmon\B99D61D874728EDC0918CA0EB10EAB93D381E7367E377406E65963366C874450.exe \Sysmon\B99D61D874728EDC0918CA0EB10EAB93D381E7367E377406E65963366C874450.exe \Users\Public\NVISO.exe
The presence of hard links within the Sysmon archive can cause an edge-case should the non-archive path be locked by another process while we attempt to clean the archive. Should for example a process be created from the non-archive path, removing the archived file will become slightly harder.
:: If the other path is locked by a process, deleting it will result in a denied access. C:\>del Sysmon\B99D61D874728EDC0918CA0EB10EAB93D381E7367E377406E65963366C874450.exe C:\Sysmon\B99D61D874728EDC0918CA0EB10EAB93D381E7367E377406E65963366C874450.exe Access is denied.
Removing hard links is not straight-forward and commonly relies on non-native software such as fsutil
(itself requiring the Windows Subsystem for Linux). However, as the archive’s hard link does technically not consume additional storage (the same content is referenced from another path), such files could be ignored given they do not partake in the storage exhaustion. Once the non-archive hard links referencing a Sysmon-archived file are removed, the archived file is not considered a hard link anymore and will be removable again.
To cope with the above edge-case, hard links can be filtered-out and removal operations can be encapsulated in try/catch expressions should other edge-cases exists. Overall, the WMI logical consumer’s logic could look as follow:
$Archived = Get-ChildItem -Path 'C:\\Sysmon\\' -File | Where-Object {$_.LinkType -ne 'HardLink'} | Sort-Object -Property LastAccessTimeUtc $Size = ($Archived | Measure-Object -Sum -Property Length).Sum for($Index = 0; ($Index -lt $Archived.Count) -and ($Size -gt 5GB); $Index++) { try { $Archived[$Index] | Remove-Item -Force -ErrorAction Stop $Size -= $Archived[$Index].Length } catch {} }
As we did for the event filter, a WMI consumer can be created through the New-CimInstance
cmdlet. The following snippet specifically creates a new CommandLineEventConsumer
invoking our above clean-up logic to create a 10GB quota.
$Archive = "C:\\Sysmon\\" $Limit = 10GB $Consumer = New-CimInstance -Namespace root/subscription -ClassName CommandLineEventConsumer -Property @{ Name = 'SysmonArchiveCleaner'; ExecutablePath = $((Get-Command PowerShell).Source); CommandLineTemplate = "-NoLogo -NoProfile -NonInteractive -WindowStyle Hidden -Command `"`$Archived = Get-ChildItem -Path '$Archive' -File | Where-Object {`$_.LinkType -ne 'HardLink'} | Sort-Object -Property LastAccessTimeUtc; `$Size = (`$Archived | Measure-Object -Sum -Property Length).Sum; for(`$Index = 0; (`$Index -lt `$Archived.Count) -and (`$Size -gt $Limit); `$Index++){ try {`$Archived[`$Index] | Remove-Item -Force -ErrorAction Stop; `$Size -= `$Archived[`$Index].Length} catch {}}`"" }
In the above two sections we defined the event filter and logical consumer. One last point worth noting is that event filters need to be bound to an event consumers in order to become operational. This is done through a __FilterToConsumerBinding
instance as shown below.
New-CimInstance -Namespace root/subscription -ClassName __FilterToConsumerBinding -Property @{ Filter = [Ref]$Filter; Consumer = [Ref]$Consumer; }
The following proof-of-concept deployment technique has been tested in limited environments. As should be the case with anything you introduce into your environment, make sure rigorous testing is done and don’t just deploy straight to production.
The following PowerShell script creates a WMI event filter and logical consumer with the logic we defined previously before binding them. The script can be configured using the following variables:
$Archive
as the Sysmon archive path. To be WQL-compliant, special characters have to be back-slash (\
) escaped, resulting in double back-slashed directory separators (\\
).$Limit
as the Sysmon archive’s desired maximum folder size (see real literals).$Delay
as the event filter’s maximum WQL delay value in seconds (WITHIN
clause).Do note that Windows security boundaries apply to WMI as well and, given the Sysmon archive directory is restricted to the SYSTEM
user, the following script should be ran using the SYSTEM
privileges.
$ErrorActionPreference = "Stop" # Define the Sysmon archive path, desired quota and query delay. $Archive = "C:\\Sysmon\\" $Limit = 10GB $Delay = 10 # Create a WMI filter for files being created within the Sysmon archive. $Filter = New-CimInstance -Namespace root/subscription -ClassName __EventFilter -Property @{ Name = 'SysmonArchiveWatcher'; EventNameSpace = 'root\cimv2'; QueryLanguage = "WQL"; Query = "SELECT * FROM __InstanceCreationEvent WITHIN $Delay WHERE TargetInstance ISA 'CIM_DataFile' AND TargetInstance.Drive='$(Split-Path -Path $Archive -Qualifier)' AND TargetInstance.Path='$(Split-Path -Path $Archive -NoQualifier)' GROUP WITHIN $Delay" } # Create a WMI consumer which will clean up the Sysmon archive folder until the quota is reached. $Consumer = New-CimInstance -Namespace root/subscription -ClassName CommandLineEventConsumer -Property @{ Name = 'SysmonArchiveCleaner'; ExecutablePath = (Get-Command PowerShell).Source; CommandLineTemplate = "-NoLogo -NoProfile -NonInteractive -WindowStyle Hidden -Command `"`$Archived = Get-ChildItem -Path '$Archive' -File | Where-Object {`$_.LinkType -ne 'HardLink'} | Sort-Object -Property LastAccessTimeUtc; `$Size = (`$Archived | Measure-Object -Sum -Property Length).Sum; for(`$Index = 0; (`$Index -lt `$Archived.Count) -and (`$Size -gt $Limit); `$Index++){ try {`$Archived[`$Index] | Remove-Item -Force -ErrorAction Stop; `$Size -= `$Archived[`$Index].Length} catch {}}`"" } # Create a WMI binding from the filter to the consumer. New-CimInstance -Namespace root/subscription -ClassName __FilterToConsumerBinding -Property @{ Filter = [Ref]$Filter; Consumer = [Ref]$Consumer; }
Once the WMI event consumption configured, the Sysmon archive folder will be kept at reasonable size as shown in the following capture where a 90KB quota has been defined.
With Sysmon archiving under control, we can now happily wait for new attacker tool-kits to be dropped…