Build, install and integrate Sysmon for Linux (Mint, Ubuntu 21, x86-64 VM)
Sysmon for Linux is a system diagnostics tool, which allows trace and log analysis for system and security engineering. It provides a systematic and specified output, allowing automated data processing for analytics tasks at scale.
Note that there is a lot of false information published on the internet regarding logging, and how Blue Teams / defenders perform log analysis. Storing a massive amount of data will not always enable anyone to detect malicious activity. You need to be selective about the sources.
Sysmon is a valuable source for an active defense strategy, but it’s also very verbose. This may only be suitable for select use cases.
The following article assumes intermediate or higher Linux security exp.
Sysmon for Linux
Sysmon for Linux is an eBPF based tool chain for system-level tracing.
eBPF is a Dtrace-inspired framework for
troubleshooting,
root-cause analysis,
performance engineering and
finally: security monitoring.
It’s used in newer Linux kernels, potentially depending on the Linux distribution vendor settings. Older CentOS Stream and / or RHEL kernels have eBPF disabled. Building custom kernels isn’t always an option.
strace etc. vs Sysmon
Sysmon for Linux offers more structured reporting implementation, allowing automated data-analysis. On Linux specifically, there is an abundance of diagnostics and monitoring tools. Sysmon stands out, because it uses eBPF and therefore has a lower performance impact than others.
Revised build documentation for Linux (Mint)
If you only install Sysmon from the package manager, you will miss the sysmonLogView tool. Besides that, not all Linux systems have readily available Sysmon builds.
Prepare the SysinternalsEBPF build environment
As already indicated, Sysmon uses eBPF.
SysinternalsEBPF attempts to automatically discover the offsets of some members
of some kernel structs. If this fails, please provide details of the kernel
version (and config if possible) plus the error message to the GitHub issues page.
SysinternalsEBPF comprises two parts:
a shared library that userland programs can link with;
and an eBPF code library that eBPF programs can include to interact successfully with the library.
What this does is, it allows Sysmon to subscribe to eBPF telemetry data. This library is GPL / LGPL licensed (my Microsoft).
Build the lib is a straightforward process on a DEB-based Linux:
sudo apt update
sudo apt -y install build-essential gcc g++ make cmake libelf-dev llvm clang libzstd1 git libjson-glib-dev libc6-dev-i386
git clone https://github.com/Sysinternals/SysinternalsEBPF.git
cd Sysmon/sysinternalsEBPF
mkdir build
cd build
cmake ..
make
Prepare the Sysmon build environment with .Net 6 on Linux
The Sysmon build documentation for Linux misses a couple of crucial points. The official build document can be found here.
Sysmon is a userland application (which will be run as an elevated process). It has got a Systemd integration (unit file, which will load the conifg later).
Based on the offical docs, you can run:
marius@mleng:~/$ mkdir ~/source/dotnet
marius@mleng:~/$ wget https://dot.net/v1/dotnet-install.sh -O dotnet-install.sh
chmod +x dotnet-install.sh
# you need 6.x for sysmon
marius@mleng:~/$ ./dotnet-install.sh --channel 6.0
marius@mleng:~/$ dotnet --list-sdks
6.0.418 [/home/marius/.dotnet/sdk]
# if you have multiple SDKs, the newest will be used by default
# handling this is possible by switch SDK versions
# https://stackoverflow.com/questions/42077229/switch-between-dotnet-core-sdk-versions
# to the .profile or equivalently used dotfile for your Shell env
DOTNET_ROOT=/home/$USER/.dotnet
sudo apt update
dotnet tool install --global dotnet-t4 --version 2.3.1
sudo apt -y install build-essential gcc g++ make cmake libelf-dev llvm clang libxml2 libxml2-dev libzstd1 git libgtest-dev apt-transport-https dirmngr googletest google-mock libgmock-dev libjson-glib-dev libc6-dev-i386 libssl-dev
git clone --recurse-submodules https://github.com/Sysinternals/SysmonForLinux.git
cd SysmonForLinux
mkdir build
cd build
cmake ..
make
marius@mleng:~/source/SysmonForLinux/build$ sudo ./sysmon -i
Sysmon v0.0.0 - Monitors system events
Sysinternals - www.sysinternals.com
By Mark Russinovich, Thomas Garnier and Kevin Sheldrake
Copyright (C) 2014-2023 Microsoft Corporation
Licensed under MIT/GPLv2
Using libxml2. libxml2 is Copyright (C) 1998-2012 Daniel Veillard. All Rights Reserved.
Created symlink /etc/systemd/system/multi-user.target.wants/sysmon.service → /etc/systemd/system/sysmon.service.
marius@mleng:~/source/SysmonForLinux/build$ sudo systemctl disable sysmon
Removed /etc/systemd/system/multi-user.target.wants/sysmon.service.
marius@mleng:~/source/SysmonForLinux/build$ sudo systemctl stop sysmon
marius@mleng:~/source/SysmonForLinux/build$ ls /opt/sysmon/
ls: cannot open directory '/opt/sysmon/': Permission denied
marius@mleng:~/source/SysmonForLinux/build$ sudo !!
sudo ls /opt/sysmon/
argc eventId sysmonEBPFkern4.15_core.o sysmonEBPFkern4.16.o sysmonEBPFkern5.2_core.o sysmonEBPFkern5.3-5.5.o sysmonLogView
argv rules.bin sysmonEBPFkern4.15.o sysmonEBPFkern4.17-5.1_core.o sysmonEBPFkern5.2.o sysmonEBPFkern5.6-_core.o
config.xml sysmon sysmonEBPFkern4.16_core.o sysmonEBPFkern4.17-5.1.o sysmonEBPFkern5.3-5.5_core.o sysmonEBPFkern5.6-.o
marius@mleng:~/source/SysmonForLinux/build$
Load MSTIC configs
Sysmon’s data can only be handled if you configure it well.
The best place to look for good config… obviously Microsoft R&D.
Let’s check it:
Oki, the config file got copied over.
Patch the Systemd unit
If you inspect that file, you’ll see that it looks pretty bare bones.
Let’s add:
CPU / memory / IO limitation
a journal ring buffer limit
Create a file and name it sysmon.conf . Occasionally, systemd likes to undo your changes, which can be extremely annoying. Don’t use “override files” or any advanced features of systemd. None of them are documented well, or frequently tested. The trick is not to use them.
And copy this unit file over:
Great, we have tamed a system-level tracer and limited its potential resource consumption. This way, the system will be more reliable. Log writing can cause resource exhaustion on busy systems. We don’t want to disrupt mission-critical systems. And this is a way to prevent this, if the load gets high.
Log handling
We also need to ensure that the disks don’t fill up. On hardened systems (following the CIS benchmark baselines, for example), /var/log will be a separate partition. This is very recommended for production.
Generally, you can take a look at the concept from the perspective of RHEL here:
The traditional Linux way of limiting log file sizes is a combination of Rsyslog and logrotate.
/etc/rsyslog/66-rsyslog.conf
/etc/logrotate.d/sysmon
Check that the configs were accepted:
This looks good.
A ring buffer for the file(s)
I took a look at circFS, which promises to create a circular disk-space limited ringbuffer, like a FIFO queue.
First of all, I created a Makefile and installed libfuse-dev.
Sysmon writes the logs as an elevated root process.
But the mount point dies consistently.
I am looking for something better, that does depend on endless Bash loops or FUSE.
Sysmon logs on Linux are only XML-formatted
The logs in /var/log/sysmon.log will be XML, prefixed with Syslog timestamps:
They have got their own TimeCreated key (in UTC). That means that the additional Syslog timestamp can be ignored here. Sysmon doesn’t offer an option on Linux (as far as I know) to write XML into a separate file. Rsyslog will do that for is, but generally this isn’t a nice and clean setup.
The log format isn’t great:
an additional syslog timestamp
no XML validation for all I know
special characters get escaped (<, >) (which makes sense in XML)
XML isn’t the friendliest markup these days
Log investigation with the command line (XML to JSON)
Transform XML records to JSON (multi-threaded)
The file /var/log/sysmon.log is being used by our Sysmon service.
In the following, the file is concurrently being processed with some modern log investigation tools. You can archive the same with awk, grep etc. Multi-threaded or concurrency doesn’t necessarily speed up these tasks, but I have some log data enrichment tasks in mind where it may be of use (later).
Python Sysmon XML to JSON transformer
This transformer uses:
json (standard Python 3)
LXML
ThreadPoolExecutorbecause LXML depends on cPython (serialization limitations apply)
Use ripgrep and jq
If you have multiple files (maybe due to logrotate), ripgrep can provide a concurrency enabled search workflow. We are parameterizing it to extract the Event element from the XML line:
You can remove the -m parameter, and handle the tasks separately.
If you expand the example, you’ll see that Sysmon provides a rich hierarchical data set, ordered for automated analysis. The data has been transformed to JSON.
Summary
Sysmon build documented (hopefully it helps others), extended with the .Net dependency
Sysmon integration documented, addressing resource limits and Systemd configuration
basic recap with logrotate and Rsyslog
quick check on circFS (looking for alternatives, obviously)
Sysmon security config from MSTIC referenced, referring to security focused system monitoring and observation specification
System data analysis documented, with a custom (modern) tool chain and basic Python code. This can be used for data enrichment / research / understanding the concepts
Sysmon XML to JSON transformer published