{"type":"doc","content":[{"type":"paragraph"},{"type":"blockCard","attrs":{"url":"https://www.because-security.com/blog/the-end-of-free-tier-esxi"}},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"My personal 2024 setup is simple and effective:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"fast, Free Open-Source","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"It’s ","type":"text"},{"text":"not","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"flawless because it’s not funded well","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"or easy to learn because it’s not well documented","type":"text"}]}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"And it’s not as fast as ESXi, specifically for Windows guests (depends on the kind of workload):","type":"text"}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"children","parameters":{"macroParams":{"allChildren":{"value":"true"}},"macroMetadata":{"macroId":{"value":"9811d411-cd34-4a9e-b5f9-654640047a1d"},"schemaVersion":{"value":"2"},"title":"Child pages (Children Display)"}},"localId":"34cbce60-f7aa-4e82-95ac-3b25db492554"}},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"Read the summary if you are evaluating this for Windows guests. ","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"– I don’t believe VMware products will benefit from the 2023 Broadcom acquisition. Let’s learn from history.","type":"text"}]},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"text":"Announcement","type":"text","marks":[{"type":"strong"}]},{"text":": End of general availability of ESXi (free tier). ","type":"text"},{"type":"date","attrs":{"timestamp":"1707782400000"}},{"text":" ","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://kb.vmware.com/s/article/2107518"}},{"type":"blockCard","attrs":{"url":"https://twitter.com/windsheep_/status/1757371562016059719"}},{"type":"paragraph","content":[{"text":" ","type":"text"},{"text":"Solution","type":"text","marks":[{"type":"strong"}]},{"text":": move on. ","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"The alternative:","type":"text","marks":[{"type":"strong"}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Libvirt is a control software which also supports Linux KVM guests (based on qemu). – A simple hypervisor (KVM) with simple tooling (qemu etc.). Many opportunities.","type":"text"}]},{"type":"paragraph","content":[{"text":"To name a few:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Windows 11 remote clients (TPM 2.0) ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-check_mark","text":":check_mark:","shortName":":check_mark:"}},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Ubuntu 22 LTS Server for self-hosting ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-check_mark","text":":check_mark:","shortName":":check_mark:"}}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Terabytes of Cloud Storage, managed via Shared Folders (Virtio) ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-check_mark","text":":check_mark:","shortName":":check_mark:"}},{"text":" ","type":"text"}]}]}]}]},{"type":"paragraph","content":[{"text":"And no: I wouldn’t run a data-center with this. Maybe with oVirt, but not with plain Libvirt Linux KVM.","type":"text"},{"type":"hardBreak"},{"text":"And yes: you can run any system you want if it’s supported. But I have not tested that.","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"The setup in the following:","type":"text","marks":[{"type":"strong"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":" It’s a single-host setup. This is for small-scale scenarios.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"1 Gpbs, no LACP, no nothing","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"RAID 1, no HBA, no nothing","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"no vLANs, no vSwitch, no nothing","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"…","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"you get the point: ","type":"text"},{"text":"small-scale","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"no VM at-rest encryption (on Hypervisor-management level because libvirt has a useless secrets handling process)","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"you can use luks in qcow2 images, but you need to store the key in cleartext ","type":"text"},{"type":"emoji","attrs":{"id":"1f427","text":"🐧","shortName":":penguin:"}},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"of course you can use Full-Disk Encryption on the guests","type":"text"}]}]}]}]}]}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"ToC","type":"text","marks":[{"type":"strong"}]}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{},"macroMetadata":{"macroId":{"value":"e23672e5-4f25-4067-8fa9-48a671b83c43"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"b7e5fc56-8786-45a8-8971-61f2010bf9b2"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Compartmentalize the HW host into many virtual guests","type":"text"}]},{"type":"paragraph","content":[{"text":"In the following, I will only write about the interesting stuff.","type":"text"}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"drawio-sketch","parameters":{"macroParams":{"mVer":{"value":"2"},"zoom":{"value":"1"},"simple":{"value":"0"},"inComment":{"value":"0"},"custContentId":{"value":"43942275"},"pageId":{"value":"42336311"},"lbox":{"value":"1"},"diagramDisplayName":{"value":"kvm-qemu-host"},"contentVer":{"value":"1"},"revision":{"value":"1"},"baseUrl":{"value":"https://because-security.atlassian.net/wiki"},"diagramName":{"value":"kvm-qemu-host"},"pCenter":{"value":"0"},"width":{"value":"748"},"links":{"value":""},"tbstyle":{"value":""},"height":{"value":"511"}},"macroMetadata":{"macroId":{"value":"9cf7305a-9c0a-49d8-9df8-d1462eed7455"},"schemaVersion":{"value":"1"},"placeholder":[{"type":"image","data":{"url":"https://ac.draw.io/connectImage?mVer=2&zoom=1&simple=0&inComment=0&custContentId=43942275&pageId=42336311&lbox=1&diagramDisplayName=kvm-qemu-host&contentVer=1&revision=1&baseUrl=https%3A%2F%2Fbecause-security.atlassian.net%2Fwiki&diagramName=kvm-qemu-host&pCenter=0&width=748&links=&tbstyle=&height=511"}}],"title":"draw.io Board"}},"localId":"fca5c553-6c12-4f48-ad77-143c68b280e4"}},{"type":"paragraph","content":[{"text":"Simple is better. NAT mode, ","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"You can map guests out directly with a Zero Trust SASE architecture:","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://because-security.atlassian.net/wiki/spaces/AD/pages/43286550"}},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"Or you use iptables for NATing, if you need higher throughput / reliability. I use both. This is a small-scale setup without an LB / big CDN cache etc. ","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Dedicated Server","type":"text"}]},{"type":"paragraph"},{"type":"blockCard","attrs":{"url":"https://www.youtube.com/watch?v=5eo8nz_niiM"}},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"text":"Hetzner Serverbörse ","type":"text"},{"type":"emoji","attrs":{"id":"1f642","text":"🙂","shortName":":slight_smile:"}},{"text":" 50 bucks, 8 TB cloud storage + computation is dirt cheap in 2024.","type":"text"}]},{"type":"table","attrs":{"layout":"default","width":760.0,"localId":"24bd0bc2-3b1e-47c9-8323-618d08e1d78c"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"HW","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"Spec","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"Purpose","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"CPU","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"i7 8700 ","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"12 cores for small-scale VMs","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"RAM","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"128 GB","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"split between VMs","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"Disk","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"8 TB","type":"text"}]},{"type":"paragraph","content":[{"text":"RAID 1","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"2 TB - base OSes","type":"text"}]},{"type":"paragraph","content":[{"text":"6 TB - data","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"Net","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"IPv4","type":"text"}]},{"type":"paragraph","content":[{"text":"IPv6","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"routeable IPs","type":"text"}]},{"type":"paragraph","content":[{"text":"NAT internal net to external IP:Port","type":"text"}]}]}]}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Headless setup (remote): libvirt daemon","type":"text"}]},{"type":"paragraph","content":[{"text":"Install: SSH connect to the HW server (Debian 12 Bookworm):","type":"text"}]},{"type":"codeBlock","content":[{"text":"apt install --no-install-recommends qemu-system libvirt-clients libvirt-daemon-system\napt install qemu-block-extra qemu-utils # qcow2 format support, block optional\napt install ovmf # UEFI\napt install swtpm-tools swtpm-utils # tpm 2.0\napt install dnsmasq # default network 192.168/24, NAT via IPtables\nsystemctl disable dnsmasq # libvirt only needs the util, not the service\n\n# optionally\napt install irqbalance\napt install haveged\n# apt install numad # on different HW\napt install apparmor-profiles","type":"text"}]},{"type":"paragraph","content":[{"text":"TPM emulation is for Windows 11.","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"TPM (2.0) is not a security feature. It’s a way to ensure that corporate systems with Full-Disk Encryption can be administered by IT. TPMs may have a reasonable secure at-rest encryption. But no in-transit encryption.","type":"text"}]},{"type":"paragraph","content":[{"text":"Given that the TPM transmits the Bitlocker Key in clear-text (via the service here, or via the Bus on HW systems), it undermines the security of Bitlocker. Use a different encryption, for example LVM on the host with dmcrypt LUKS (without TPM) or an encrypted qcow2 file backend. ","type":"text"}]}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"Dedicated server: KSM (Kernel Same-page Merging)","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"Kernel Same-page Merge (KSM) is a process that allows guests to share identical memory pages. By sharing pages, the combined memory usage of the guest is reduced. The savings are especially increased when multiple guests are running similar base operating system images.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"(","type":"text"},{"text":"Source","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.ibm.com/support/knowledgecenter/en/linuxonibm/liabp/liabpksm.htm"}}]},{"text":")","type":"text"}]},{"type":"paragraph","content":[{"text":"Many Linux distributions do not enable KSM by default, even if you compile the kernel feature. The ","type":"text"},{"text":"ksmtuned","type":"text","marks":[{"type":"code"}]},{"text":" daemon can dynamically adjust the frequency of the deduplication algorithms.","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://github.com/ksmtuned/ksmtuned"}},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo apt-get install ksmtuned --no-install-recommends","type":"text"}]},{"type":"paragraph","content":[{"text":"Or manually (older example, from around 2015):","type":"text"}]},{"type":"codeBlock","content":[{"text":"root@mjo:/home/marius/scripts# grep -H '' /sys/kernel/mm/ksm/pages_*\n/sys/kernel/mm/ksm/pages_shared:0\n/sys/kernel/mm/ksm/pages_sharing:0\n/sys/kernel/mm/ksm/pages_to_scan:100\n/sys/kernel/mm/ksm/pages_unshared:0\n/sys/kernel/mm/ksm/pages_volatile:0\nroot@mjo:/home/marius/scripts# cat /sys/kernel/mm/ksm/run\n0\nroot@mjo:/home/marius/scripts# echo 1 > /sys/kernel/mm/ksm/run\nroot@mjo:/home/marius/scripts# grep -H '' /sys/kernel/mm/ksm/pages_*\n/sys/kernel/mm/ksm/pages_shared:0\n/sys/kernel/mm/ksm/pages_sharing:0\n/sys/kernel/mm/ksm/pages_to_scan:100\n/sys/kernel/mm/ksm/pages_unshared:0\n/sys/kernel/mm/ksm/pages_volatile:15900\n","type":"text"}]},{"type":"paragraph","content":[{"text":"What exactly the kernel will do with my 15900 volatile pages depends on the kernel version. If you set it this, the value will grow first, and then at some point residual pages can be shared. ","type":"text"},{"text":"Here are more details","type":"text","marks":[{"type":"link","attrs":{"href":"https://www.ibm.com/developerworks/library/l-kernel-shared-memory/"}}]},{"text":" about this.","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"pages_shared: The number of unswappable kernel pages that KSM is using","type":"text"},{"type":"hardBreak"},{"text":"pages_sharing: An indication of memory savings","type":"text"},{"type":"hardBreak"},{"text":"pages_unshared: The number of unique pages repeatedly checked for merging","type":"text"},{"type":"hardBreak"},{"text":"pages_volatile: The number of pages that are changing too often","type":"text"}]}]},{"type":"paragraph","content":[{"text":"The bottom line is: KSM can share pages between guests, which do not change frequently. So if you check 15 minutes later, it may look like this:","type":"text"}]},{"type":"paragraph","content":[{"type":"date","attrs":{"timestamp":"1707782400000"}},{"text":" ","type":"text"}]},{"type":"codeBlock","content":[{"text":"root@Debian-bookworm-latest-amd64-base ~ # grep -H '' /sys/kernel/mm/ksm/pages_*\n/sys/kernel/mm/ksm/pages_shared:13\n/sys/kernel/mm/ksm/pages_sharing:1341\n/sys/kernel/mm/ksm/pages_to_scan:100\n/sys/kernel/mm/ksm/pages_unshared:336\n/sys/kernel/mm/ksm/pages_volatile:0\n","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"KSM is efficient, especially if you base your VMs on templates. ","type":"text"}]},{"type":"table","attrs":{"layout":"default","width":760.0,"localId":"7a2fec22-fa97-42f7-a2ea-8fb83d82f275"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"VMs (shared memory)","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"RAM","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"Used (KSM) on the host","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"3 x Win 11 ","type":"text"},{"type":"hardBreak"},{"text":"(Template)","type":"text"}]},{"type":"paragraph"}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"12 GB","type":"text"}]},{"type":"paragraph","content":[{"text":"12 GB","type":"text"}]},{"type":"paragraph","content":[{"text":"12 GB","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":2},"content":[{"type":"paragraph","content":[{"text":"Transparent Huge Pages Support ","type":"text"}]},{"type":"paragraph","content":[{"text":"KSM daemon","type":"text"}]},{"type":"paragraph","content":[{"text":"Debian 12 host","type":"text"}]},{"type":"paragraph","content":[{"text":"Template Machines","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"6 x Ubuntu Server 22.04 LTS 22","type":"text"},{"type":"hardBreak"},{"text":"(Template)","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"12 GB","type":"text"}]},{"type":"paragraph","content":[{"text":"4 GB","type":"text"}]},{"type":"paragraph","content":[{"text":"4 GB","type":"text"}]},{"type":"paragraph","content":[{"text":"8 GB","type":"text"}]},{"type":"paragraph","content":[{"text":"8 GB","type":"text"}]},{"type":"paragraph","content":[{"text":"4 GB","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"Sum","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"76 GB","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1},"content":[{"type":"paragraph","content":[{"text":"46,6","type":"text","marks":[{"type":"strong"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"Roughly saves 25 GB, 30% here. ","type":"text"}]},{"type":"paragraph"},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"tuned - the right settings for the kernel, at the right time","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://tuned-project.org/"}},{"type":"paragraph","content":[{"text":"Tuned belongs to the standard Red Hat libvirt KVM stack. You should also add it to you guests, with the guest profile, to avoid swappiness of database hosts.","type":"text"}]},{"type":"codeBlock","content":[{"text":"apt install tuned","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"And in the crontab:","type":"text"}]},{"type":"codeBlock","content":[{"text":"1 0 * * * /usr/sbin/tuned-adm profile server-powersave\n1 8 * * * /usr/sbin/tuned-adm profile virtual-host","type":"text"}]},{"type":"paragraph","content":[{"text":"This way we can save power at night.","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"Debian 12 has Transparent Huge Pages support","type":"text"}]},{"type":"paragraph","content":[{"text":"We don’t need to adjust this:","type":"text"}]},{"type":"codeBlock","content":[{"text":"root@Debian-bookworm-latest-amd64-base /var/log # cat /proc/meminfo | grep HugePages\nAnonHugePages: 18608128 kB\nShmemHugePages: 0 kB\nFileHugePages: 0 kB\nHugePages_Total: 0\nHugePages_Free: 0\nHugePages_Rsvd: 0\nHugePages_Surp: 0\nroot@Debian-bookworm-latest-amd64-base /var/log # cat /sys/kernel/mm/transparent_hugepage/enabled\n[always] madvise never\n\nroot@Debian-bookworm-latest-amd64-base /var/log # cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size\n2097152\nroot@Debian-bookworm-latest-amd64-base /var/log #","type":"text"}]},{"type":"paragraph","content":[{"text":"The kernel takes care of this automatically. ","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"Firewall - iptables for IPv4 and IPv6","type":"text"}]},{"type":"paragraph","content":[{"text":"In some cases, dnsmasq (controlled by libvirt) may open DHCP and DNS listening sockets:","type":"text"}]},{"type":"codeBlock","content":[{"text":"root@Debian-bookworm-latest-amd64-base /var/log # ss -lpn 'sport = :53 or sport = :67'\nNetid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process\nudp UNCONN 0 0 192.168.122.1:53 0.0.0.0:* users:((\"dnsmasq\",pid=1080,fd=5))\nudp UNCONN 0 0 0.0.0.0%virbr0:67 0.0.0.0:* users:((\"dnsmasq\",pid=1080,fd=3))\ntcp LISTEN 0 32 192.168.122.1:53 0.0.0.0:* users:((\"dnsmasq\",pid=1080,fd=6))","type":"text"}]},{"type":"paragraph","content":[{"text":"Here we see a DHCP listening socket bound to external interfaces. In case, you also see a public DNS resolver, you may also want to adjust your IPtables rules for that.","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\niptables -A INPUT -i eno1 -p tcp --dport 53 -j DROP\niptables -A INPUT -i eno1 -p udp --dport 53 -j DROP\n\nip6tables -A INPUT -i eno1 -p tcp --dport 53 -j DROP\nip6tables -A INPUT -i eno1 -p udp --dport 53 -j DROP\n\niptables -A INPUT -i eno1 -p udp --dport 67 -j DROP\niptables -A INPUT -i eno1 -p tcp --dport 67 -j DROP\n\nip6tables -A INPUT -i eno1 -p udp --dport 67 -j DROP\nip6tables -A INPUT -i eno1 -p tcp --dport 67 -j DROP\n","type":"text"}]},{"type":"paragraph","content":[{"text":"You may want to send RST instead. ","type":"text"}]},{"type":"paragraph","content":[{"text":"If this is a perimeter-facing device, ","type":"text"},{"text":"conntrack","type":"text","marks":[{"type":"code"}]},{"text":" may be useful to you.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Guest IP enumeration ","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://www.libvirt.org/manpages/virsh.html"}},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"text":"virsh","type":"text","marks":[{"type":"code"}]},{"text":" isn’t able to show a list like this:","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"Guest hostname - Guest IP","type":"text"}]}]},{"type":"paragraph","content":[{"text":"PowerCLI can. For libvirt, you read that right. I haven’t used that in years, but it’s an option.","type":"text"}]},{"type":"paragraph"},{"type":"blockCard","attrs":{"url":"https://github.com/nyanhp/PowerShellLibVirt"}},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Dnsmasq leases ","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://thekelleys.org.uk/dnsmasq/doc.html"}},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"text":"Assuming that you use DHCP, you will probably have ","type":"text"},{"text":"dnsmasq","type":"text","marks":[{"type":"code"}]},{"text":" on the host machine. Now what are the IPs of the guest VMs? I want to SSH / RDP into my web server VM.","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\nLEASES=/var/lib/libvirt/dnsmasq/virbr0.leases\n\nif [ -z $1 ]; then\n echo -e \"\\e[00;31mNo argument given, chose hostname from\\e[00m:\"\n cat $LEASES | awk '{ print $3 \" \" $4 }'\n exit 2\nfi\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Result (example):","type":"text"}]},{"type":"codeBlock","content":[{"text":"192.168.100.10 web\n192.168.100.20 teamcity\n192.168.100.30 gitlab\n192.168.100.40 ansible-tower\n192.168.100.50 docker-containers\n...\n","type":"text"}]},{"type":"paragraph","content":[{"text":"This assumes:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"virbr0","type":"text","marks":[{"type":"code"}]},{"text":" is your bridge interface","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"you use ","type":"text"},{"text":"dnsmasq","type":"text","marks":[{"type":"code"}]},{"text":" on Debian 12 (paths can differ)","type":"text"}]}]}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":4},"content":[{"text":"Bash and virsh","type":"text"}]},{"type":"paragraph","content":[{"text":"Same result, different script, just ","type":"text"},{"text":"virsh","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\nfor vm in $(virsh list --name); do echo \"VM name: $vm\"; virsh domifaddr $vm; done\n\nVM name: filer\nsetlocale: No such file or directory\n Name MAC address Protocol Address\n-------------------------------------------------------------------------------\n vnet1 52:54:00:df:eb:b1 ipv4 192.168.122.203/24\n\nVM name: sec-eng\nsetlocale: No such file or directory\n Name MAC address Protocol Address\n-------------------------------------------------------------------------------\n vnet2 52:54:00:4e:13:84 ipv4 192.168.122.73/24\n","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"How to NAT out a KVM guest via iptables","type":"text"}]},{"type":"paragraph","content":[{"text":"By default, my KVM guests get an internal IP (version 4). I use iptables NAT, and port forwarding to map the specific service to the external interface.","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\n# enable IP forwarding \necho \"1\" > /proc/sys/net/ipv4/ip_forward\n\n# enable the stateful firewall (for NAT)\nmodprobe ip_conntrack\n\n# we want to flush NAT \niptables -t nat -F \n\nGUEST_IP=192.168.123.123\nGUEST_PORT=22\nHOST_PORT=2223\niptables -I FORWARD -o virbr0 -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT\niptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT\n\n# Optional\n# disables conntrack for traffic \niptables -t raw -A PREROUTING -d $GUEST_IP -j NOTRACK\niptables -t raw -A OUTPUT -d $GUEST_IP -j NOTRACK\n\n# Optional\n# masquerades the VM's internal IP address to the host's IP address\nsudo iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE","type":"text"}]},{"type":"paragraph","content":[{"text":"You can automize this:","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://wiki.libvirt.org/Networking.html#Forwarding_Incoming_Connections"}},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"I took a look at Netfilter ","type":"text"},{"text":"nftables","type":"text","marks":[{"type":"code"}]},{"text":" but the core problem is the same: the organization of NAT, prerouting, filter and other chains is a mess. Pf is a much better system.","type":"text"}]}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"This assumes:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"clean NAT tables","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"virbr0","type":"text","marks":[{"type":"code"}]},{"text":" being used for this guest","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph"}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"IP forwarding is not active by default on your distribution","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"you want to use stateful iptables rules, but not everywhere","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"guests will initiate connections (you probably don’t need to do this)","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Add whatever services you like.","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"Using vhost-net and bbr with KVM qemu guests for better speeds (1 Gbps)","type":"text"}]},{"type":"paragraph","content":[{"text":"I have to use Virtio and NAT, because of the Hetzner networks. You’d need to be careful with the vSwitch topology, to avoid that virtual MAC addresses are leaking ","type":"text"},{"type":"emoji","attrs":{"id":"1f58a","text":"🖊️","shortName":":pen_ballpoint:"}},{"text":" ","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"With VMware vSwitch you can use two Layer 2 bridges, but I wanted to get rid of this complexity. In theory, however, you can archive the same with OpenVSwitch. But why bother, if you only have one single NIC and a 1 Gbps uplink. No LACP, not vLANs etc.","type":"text"}]},{"type":"paragraph","content":[{"text":"For reference:","type":"text","marks":[{"type":"strong"}]}]},{"type":"blockCard","attrs":{"url":"https://because-security.atlassian.net/wiki/spaces/ST1/pages/10027027/VMware+ESXi+for+the+lab#vSwitch-topology"}},{"type":"paragraph"},{"type":"paragraph"},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"With the following setup, I was able to push the RX throughput from 40 MB/s to approx. 114 MB/s. That’s a significant improvement.","type":"text"}]}]},{"type":"mediaSingle","attrs":{"layout":"center","width":703,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":703,"alt":"Screenshot 2024-02-16 at 16.11.21.png","id":"2b694910-65d5-4c4f-b906-2d05b00cd625","collection":"contentId-42336311","type":"file","height":281}},{"type":"caption","content":[{"text":"916 Mibps RX (KVM guest activity), btop shows the “Top” value. vhost_net accelerates Virtio paravirtualized network drivers.","type":"text"}]}]},{"type":"blockCard","attrs":{"url":"https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net"}},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The brr TCP stack is the default in Debian 12.","type":"text"}]}]}]},{"type":"blockCard","attrs":{"url":"https://github.com/google/bbr"}},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"vhost-net is available as well. It allows offloading some networking to the host.","type":"text"}]}]}]},{"type":"mediaSingle","attrs":{"layout":"center","width":600,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":839,"alt":"image-20240211-144711.png","id":"adc4bb84-3148-45a6-8a1e-0c3cdf708561","collection":"contentId-42336311","type":"file","height":801}},{"type":"caption","content":[{"text":"vhost-net reference documentation by Red Hat","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Enable vhost-net by default (host system kernel, Debian 12 Bookwork, tested)","type":"text"}]},{"type":"codeBlock","content":[{"text":"echo \"vhost_net\" | sudo tee -a /etc/modules-load.d/vhost_net.conf\nmodprobe vhost_net # or restart","type":"text"}]},{"type":"paragraph","content":[{"text":"Apply these tweaks to the host system (also Debian 12 of course):","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\nsysctl -w net.core.rmem_max=16777216\nsysctl -w net.core.wmem_max=16777216\nsysctl -w net.ipv4.tcp_rmem='4096 87380 16777216'\nsysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'\n\n# sysctl -w net.ipv4.tcp_congestion_control=bbr\n\n# red hat tuning manual\necho 1 > /proc/sys/net/ipv4/conf/all/arp_filter\n\n# hetzner specific recommendation\nethtool -K eno1 tso off gso off","type":"text"}]},{"type":"paragraph","content":[{"text":"Adjust the Virtio network interface of the respective guest, which is going to have a high throughput:","type":"text"}]},{"type":"codeBlock","content":[{"text":"\n \n \n \n \n \n \n

\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Note: None of these values are secret because knowing this doesn’t add any risk. This is standard Linux admin stuff: tracking bottlenecks, looking for a fix. Performance engineering ","type":"text"},{"type":"emoji","attrs":{"id":"1f609","text":"😉","shortName":":wink:"}},{"text":" ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"4 vhost-net queues - 4 vCPUs (first formula, 4 - 4)","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Windows 11 guests, and Ubuntu Server","type":"text"}]},{"type":"paragraph","content":[{"text":"Finding documentation… man. Here and there. Hidden like easter eggs all over the internet.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"TPM 2.0 - Windows 11","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":632,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":632,"alt":"Screenshot 2024-02-12 at 17.47.21.png","id":"9a5cff96-cfda-472b-bfa6-fe6e6786e12b","collection":"contentId-42336311","type":"file","height":294}},{"type":"caption","content":[{"text":"TPM Emulation in libvirt with virt-manager","type":"text"}]}]},{"type":"paragraph","content":[{"text":"If you get permissions problems during the boot (I did):","type":"text"}]},{"type":"codeBlock","content":[{"text":"sudo chown tss:tss /var/lib/swtpm-localca/.lock.swtpm-localca\nsudo chown tss:tss /var/lib/swtpm-localca/signkey.pem","type":"text"}]},{"type":"paragraph","content":[{"text":"You can start as many Windows 11 systems as you want. You can clone them, snapshot them etc..","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Hyper-V (nested virtualization) - Windows 10 / 11 - EXPERIMENTAL","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Hyper-V on KVM is a) unsupported, b) unreliable, c) experimental","type":"text"}]},{"type":"paragraph","content":[{"text":"Nested virtualization works, for WSL2, Ubuntu on Windows etc. Not recommended for serious workloads","type":"text"}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"On the i7 8700 you must ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-warning","text":":warning:","shortName":":warning:"}},{"text":" disable Model Specific Registers (MSRs) in KVM:","type":"text"}]},{"type":"codeBlock","content":[{"text":"root@Debian-bookworm-latest-amd64-base /etc/modprobe.d # cat kvm_intel.conf\noptions kvm_intel nested=1 ept=1 vpid=1 enable_apicv=1 unrestricted_guest=1","type":"text"}]},{"type":"codeBlock","content":[{"text":"root@Debian-bookworm-latest-amd64-base /etc/modprobe.d # cat kvm.conf\noptions kvm nested=1 ignore_msrs=1 halt_poll_ns=10000 report_ignored_msrs=0 min_timer_period_us=500 tdp_mmu=1","type":"text"}]},{"type":"paragraph","content":[{"text":"The other parameters are from my tuning. ","type":"text"},{"text":"ignore_msrs","type":"text","marks":[{"type":"code"}]},{"text":" matters here.","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"Otherwise, you will get into Boot loops and Blue Screen “Threat execution” errors, which are very difficult to trace. Obviously, because MSRs are involved.","type":"text"}]},{"type":"paragraph","content":[{"text":"You can disable MSRs at runtime (in the KVM kernel module on the host, Debian 12):","type":"text"}]},{"type":"codeBlock","content":[{"text":"echo \"1\" > /sys/module/kvm/parameters/ignore_msrs","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"I tested the following Hyper-V Enlightenments (QEMU feature) ( ","type":"text"},{"type":"date","attrs":{"timestamp":"1707868800000"}},{"text":" ):","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":600,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":806,"alt":"Screenshot 2024-02-12 at 17.54.14.png","id":"d10f5d8a-214b-4bf1-a777-6985f5a1f430","collection":"contentId-42336311","type":"file","height":618}},{"type":"caption","content":[{"text":"Hyper-V Enlightenments on KVM for enhanced performance","type":"text"}]}]},{"type":"paragraph","content":[{"text":"XML for virt-manager / virsh:","type":"text"}]},{"type":"codeBlock","attrs":{"language":"xml"},"content":[{"text":" \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n ","type":"text"}]},{"type":"paragraph"},{"type":"mediaSingle","attrs":{"layout":"center","width":471,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":471,"alt":"Screenshot 2024-02-12 at 17.56.56.png","id":"158c5fbe-b07d-4526-a011-07be66eccb97","collection":"contentId-42336311","type":"file","height":441}},{"type":"caption","content":[{"text":"CPU host-passthrough works for Windows 11","type":"text"}]}]},{"type":"paragraph"},{"type":"blockCard","attrs":{"url":"https://access.redhat.com/documentation/de-de/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-manipulating_the_domain_xml-cpu_model_and_topology"}},{"type":"paragraph","content":[{"text":"You can use shared memory with nested virtualization:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":760,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":793,"alt":"Screenshot 2024-02-14 at 21.56.21.png","id":"5cdec58b-7a19-486c-a6cb-cf2b8d53722c","collection":"contentId-42336311","type":"file","height":325}},{"type":"caption","content":[{"text":"Shared memory flag set for Windows 11 guest, which has support for Hyper-V in a nested virtualization setup (KVM qemu)","type":"text"}]}]},{"type":"paragraph","content":[{"text":"This way, KSM can be used (via the daemon). ","type":"text"}]},{"type":"paragraph"},{"type":"mediaSingle","attrs":{"layout":"center","width":760,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":1511,"alt":"Screenshot 2024-02-15 at 08.24.53.png","id":"19e4f2cd-071c-4ade-a6c5-2587c4d1541b","collection":"contentId-42336311","type":"file","height":978}},{"type":"caption","content":[{"text":"A Windows 11 guest (Qemu Standard PC in Device Manager) running WSL (Terminal) and the Hyper-V hypervisor with a Docker Desktop guest (Hyper-V mode). Task Manager shows “Virtualization: Enabled”. Here 6 cores were assigned, pass-through mode. Shared memory option enabled in virt-manager. ","type":"text"}]}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"Pass a RNG","type":"text"}]},{"type":"paragraph","content":[{"text":"I pass an RNG device, which will be used by Windows automatically.","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":569,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":569,"alt":"Screenshot 2024-02-18 at 17.45.07.png","id":"251edbe9-c89d-4536-abde-ba7a1aa2043b","collection":"contentId-42336311","type":"file","height":199}}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"/dev/urandom is cryptographically secure as far as I know ( ","type":"text"},{"type":"date","attrs":{"timestamp":"1708214400000"}},{"text":" )","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"the host system uses ","type":"text"},{"text":"haveged","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"VirtIO for disk and network - all guests, Windows download","type":"text"}]},{"type":"paragraph","content":[{"text":"Install VirtIO for Windows:","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers"}},{"type":"paragraph","content":[{"text":"– provided by Fedora. Performance, drivers etc.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"On Linux, Virtio is part of the default kernel config for most distributions.","type":"text"}]}]}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":4},"content":[{"text":"Bugfix: Bluescreen on Windows 10 / 11 when Virtio is the Boot Disk","type":"text"}]},{"type":"panel","attrs":{"panelType":"error"},"content":[{"type":"paragraph","content":[{"text":"This is unsupported. Virtio drivers are buggy for Windows 10, 11 and Windows Server 2022 etc. Do not use this for serious workloads. If you see Blue Screens like this, stop and rethink: is this viable?","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Windows 10 / 11 / Server often do not load the Virtio drivers during boot, which can result in Blue Screens:","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://superuser.com/questions/1057959/windows-10-in-kvm-change-boot-disk-to-virtio/1253728#1253728"}},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"text":"Bug fix Windows 11:","type":"text","marks":[{"type":"strong"}]}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"initially keep SATA for boot disk","type":"text","marks":[{"type":"strong"}]},{"text":", use unwrap if you use qcow2","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Install Virtio drivers","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"bcdedit /set \"{current}\" safeboot minimal","type":"text","marks":[{"type":"code"}]},{"text":" on a Terminal as Admin","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Attach a Virtio dummy disk","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Reboot into Safe Mode, logon from Spice QXL console","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Optional: check with Disk Manager whether the dummy disk got detected","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"bcdedit /deletevalue \"{current}\" safeboot","type":"text","marks":[{"type":"code"}]},{"text":" on Terminal as Admin","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Shutdown","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Setup the Boot disk as Virtio ","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Result: You can use Virtio for the boot disk. ","type":"text"}]},{"type":"paragraph","content":[{"text":"→ Obviously this sucks.","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":4},"content":[{"text":"fstab for virtiofs on Linux","type":"text"}]},{"type":"paragraph","content":[{"text":"You can directly share the data to the host (this is a single host setup). Otherwise, set up an NFS server or something like that.","type":"text"}]},{"type":"codeBlock","content":[{"text":"# /etc/fstab: static file system information.\n#\n# Use 'blkid' to print the universally unique identifier for a\n# device; this may be used with UUID= as a more robust way to name devices\n# that works even if disks are added and removed. See fstab(5).\n#\n# \n# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation\n/dev/disk/by-id/dm-uuid-LVM-hHj5PIoT5f70KOoGD1u1RuiApeWFn379jxwJxlSmk3wmk3E7gHEc6X5ABGFeD7Re / ext4 defaults 0 1\n# /boot was on /dev/vda2 during curtin installation\n/dev/disk/by-uuid/2fd6ca0d-d0e1-4223-9161-2092e8077f20 /boot ext4 defaults 0 1\n/swap.img none swap sw 0 0\nshare /storage/ virtiofs rw,relatime 0 0","type":"text"}]},{"type":"paragraph","content":[{"text":"There is something called “dax mode”, which I was unable to enable. It’s supposed to increase the performance somehow, but no documentation that’s worth mentioning. That’s a general problem with Virtio. ","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"Virtio 3d","type":"text"}]},{"type":"paragraph","content":[{"text":"Delete and copy-paste into virt-manager for the Display HW. Guest has to be offline.","type":"text"}]},{"type":"codeBlock","content":[{"text":"

","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"Qemu-agent ","type":"text"}]},{"type":"paragraph","content":[{"text":"- part of the Virtio Windows installer, separate package on Linux.","type":"text"}]},{"type":"blockCard","attrs":{"url":"https://wiki.qemu.org/Features/GuestAgent"}},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Snapshots - NOT for UEFI guests","type":"text"}]},{"type":"panel","attrs":{"panelType":"error"},"content":[{"type":"paragraph","content":[{"text":"Error creating snapshot: Operation not supported: internal snapshots of a VM with pflash based firmware are not supported","type":"text"}]},{"type":"paragraph","content":[{"text":"You cannot snapshot systems using UEFI with libvirt ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-warning","text":":warning:","shortName":":warning:"}}]}]},{"type":"mediaSingle","attrs":{"layout":"center","width":500,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":699,"alt":"Screenshot 2024-02-13 at 19.46.34.png","id":"f2cba2a9-39c7-4fa1-8978-2ea6336baefe","collection":"contentId-42336311","type":"file","height":838}},{"type":"caption","content":[{"text":"virt-manager error message popup: badly documented limitation. libvirt cannot create snapshots from UEFI systems such as Windows 11 or some other guests","type":"text"}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"I also believe that you cannot snapshot systems, where you use CPU host-passthrough to enable nested virtualization. Not confirmed.","type":"text"}]},{"type":"paragraph","content":[{"text":"You can use disk snapshots. Untested.","type":"text"}]},{"type":"codeBlock","content":[{"text":"virsh snapshot-create-as

--disk-only --diskspec vda,snapshot=internal","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"It is possible to create full snapshots for BIOS guests:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":633,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":998,"alt":"Screenshot 2024-02-14 at 21.43.47.png","id":"e8f62c30-32d8-430b-97ed-19b632ee5638","collection":"contentId-42336311","type":"file","height":762}},{"type":"caption","content":[{"text":"BIOS guest snapshot creation in virt-manager (sec-eng is a Linux guest). This is similar to the ESXi / VSphere snapshot manager. ","type":"text"}]}]},{"type":"mediaSingle","attrs":{"layout":"center","width":666,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":900,"alt":"Screenshot 2024-02-15 at 10.36.36.png","id":"7fbe44ed-99a0-40fd-ad25-a3ea48d9f118","collection":"contentId-42336311","type":"file","height":640}},{"type":"caption","content":[{"text":"BIOS guest snapshot creation in virt-manager (win2k22 is a Windows Server guest). Windows Server is not UEFI only yet, and doesn’t require a TPM","type":"text"}]}]},{"type":"paragraph"},{"type":"paragraph"},{"type":"heading","attrs":{"level":2},"content":[{"text":"Summary","type":"text"}]},{"type":"paragraph","content":[{"text":"In comparison to a standalone ESXi:","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"The main advantages of KVM qemu","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"less overhead ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-plus","text":":plus:","shortName":":plus:"}},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"much","type":"text","marks":[{"type":"strong"}]},{"text":" higher resource efficiency ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-plus","text":":plus:","shortName":":plus:"}},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"much","type":"text","marks":[{"type":"strong"}]},{"text":" easier download (VMware customer portals are convoluted) ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-plus","text":":plus:","shortName":":plus:"}},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"far","type":"text","marks":[{"type":"strong"}]},{"text":" easier updates ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-plus","text":":plus:","shortName":":plus:"}},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"no web management interface with tons of software vulnerabilities ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-plus","text":":plus:","shortName":":plus:"}},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"defined command-line tools like ","type":"text"},{"text":"virsh","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-plus","text":":plus:","shortName":":plus:"}},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Manageable firewall rules (","type":"text"},{"text":"iptables","type":"text","marks":[{"type":"code"}]},{"text":" are easier than ","type":"text"},{"text":"esxcli","type":"text","marks":[{"type":"code"}]},{"text":" firewall rules)","type":"text"}]}]}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":3},"content":[{"text":"The main problems of KVM qemu","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"bad documentation","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-warning","text":":warning:","shortName":":warning:"}},{"text":" ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"example: the Virtio ","type":"text"},{"text":"dax","type":"text","marks":[{"type":"code"}]},{"text":" feature has no proper documentation","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"limitations to snapshots creation and management ","type":"text","marks":[{"type":"strong"}]},{"type":"emoji","attrs":{"id":"atlassian-warning","text":":warning:","shortName":":warning:"}},{"text":" ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"no full snapshots for UEFI guests ","type":"text","marks":[{"type":"strong"}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"heterogeneous distribution of Windows utilities: ","type":"text"},{"type":"emoji","attrs":{"id":"atlassian-warning","text":":warning:","shortName":":warning:"}},{"text":" ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Spice, Virtio, TPM emulator … across RedHat, Fedora etc. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Windows 11 / 10 are second class citizens here, same for Windows Server obviously ","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Windows performance (in some workload scenarios)","type":"text","marks":[{"type":"strong"}]},{"type":"hardBreak"},{"type":"inlineCard","attrs":{"url":"https://because-security.atlassian.net/wiki/spaces/Linix/pages/54820923"}},{"text":" ","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"lack of proper unlocking of encrypted guest images or configs","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"bad secrets management in libvirt","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"one global setting ","type":"text","marks":[{"type":"strong"}]},{"text":"(in the ","type":"text"},{"text":"kvm_intel","type":"text","marks":[{"type":"code"}]},{"text":" or amd kernel module) ","type":"text"},{"text":"enables nested virtualization for all guests","type":"text","marks":[{"type":"strong"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Windows needs this for security, Linux doesn’t need this (also not for Docker)","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"no AppArmor (Mandatory Access Control) for qemu on Debian 12 Bookworm","type":"text","marks":[{"type":"strong"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"generally KVM qemu is not hardened, audited or tested for security. Cloud vendors like Google Cloud use forks.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"bad network performance with NAT mode (bridge), unless you use","type":"text"},{"text":" ","type":"text","marks":[{"type":"strong"}]},{"text":"vhost_net","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"}]}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"So far, so good. Balanced downgrade. ","type":"text"},{"text":"Don’t use this at work. ","type":"text","marks":[{"type":"strong"}]},{"type":"emoji","attrs":{"id":"1f642","text":"🙂","shortName":":slight_smile:"}},{"text":" ","type":"text"}]}],"version":1}

Browser not supported