Benchmark: Custom workload - KVM versus ESXi - vCPU

You know what’s remarkable? We live in the data-driven age, as some tech folks say. It’s all in the data. But there is little to no public data about benchmarks. Only very theoretical assumptions.

In the following, these are virtual CPU benchmarks for randomness.

Do you think Linux KVM is faster than ESXi? Test it…

1 Challenge
2 Test strategy
3 Results
4 Systems
5 Tools
- 5.1 Hyperfine - cross-platform benchmark tool for Linux, Windows 11, Server etc.
6 Data
- 6.1 Example: Windows - hyperfine - MinGW release build
- 6.2 Code
7 Next steps

Challenge

Create a basic cross-platform benchmark for virtual CPU performance

Test strategy

1000 * 1000000 Bytes of randomness get converted to string

Results

The ESXi setup is 3x faster for Windows guests, but the CPU is comparable
Linux performance is comparable

→ it’s probably related to the code as well

Systems

These CPUs have roughly the same specs. A 10% difference would have been accounted to the hardware.

	KVM	ESXi

KVM

ESXi

CPU

Intel(R) Core(TM) i7-8700 CPU

4,6 GHz

Pass-through mode

Intel(R) Xeon(R) CPU E3-1275 v6

3,8 GHz

Hypervisor

Linux KVM

6.1.0-17-amd64 Debian 12

QEMU emulator version 7.2.9 (Debian 1:7.2+dfsg-7+deb12u5)

ESXi 6.7

The rest doesn’t matter here.

Tools

Basic command line tools have been used.

Hyperfine - cross-platform benchmark tool for Linux, Windows 11, Server etc.

https://github.com/sharkdp/hyperfine

C code listed below
Compilers referenced

Data

Feb 17, 2024

System	Time (s) MinGW / GCC 13	Time (s) MSVC 17
Win 11 (KVM guest)	56	73
Win 11 Hyper-V (WSL Ubuntu 22.04 LTS) (KVM Nested guest)	26
Ubuntu 22.04 LTS (KVM guest)	24
Arch (KVM guest)	23
Win 10 (ESXi guest)	21	28
Ubuntu 22.04 (ESXi guest)	18

Example: Windows - hyperfine - MinGW release build

Example: Windows S1 - 4 vCPUs

Release builds have been used
- Visual Studio 2022 default
- Clion default (cmake)

Code

Next steps

None.

Hyperfine is a good test tool.

I would recommend testing the performance for Windows, before moving from VMware ESXi to KVM. I was surprised. It appears to be very dependent on the type of workload. ESXi may have more optimizations.
Given that a nested Hyper-V outperformed the Windows 11 guest, I’d day that there are Windows-specific issues to be sorted out between the code, the hypervisor and the libvirt stack. I am not that interested.