# vfio-native Make a KVM guest present a hardware profile that is self-consistent with real silicon and real firmware, and measure the result against an independent library. A lot of software behaves differently, or refuses to run at all, once it can tell it is inside a hypervisor. Malware samples self-disable or take a benign branch. Commercial software takes a licensing branch. Drivers and firmware take a different code path. Any of those makes a virtual machine useless for observing what the software would actually do on a physical machine - you end up measuring the sandbox instead of the subject. Two independent problems produce that gap, and this addresses both. **KVM diverges from the AMD64 architecture** in ways a guest can read directly. The wrong exception vector for an SVM instruction at CPL>0. `#GP` where the architecture specifies `#UD`. A hypercall at CPL>0 that raises no exception at all. Those are spec-conformance defects, and one of them closes a `FIXME` the kernel already carries against itself. The three KVM patches make KVM match the architecture and nothing more, which is why they are written to be acceptable upstream. **QEMU's emulated platform is not internally consistent** with any board that was ever manufactured. ACPI link-device names no real firmware uses, PCI vendor and subsystem IDs that belong to the emulator rather than to a device, firmware table fields no shipping board sets, device identity strings that name the emulator. Correcting them makes the emulated platform look like a platform. Alongside both there is a VFIO tuning layer - vCPU pinning onto real SMT pairs, sizing the guest to one cache domain, keeping the emulator thread off the vCPU cores, host CPU isolation while the guest runs. It is independent of the fidelity work and measured separately. Scored against [VMAware](https://github.com/NotRequiem/VMAware), which runs 85 detection techniques. A stock KVM guest starts at 10/85. This gets it to **1/85** on both the v2.8.1 release and current HEAD, and the one survivor needs a real GPU passed through. VMAware's own verdict: ``` VM brand: Unknown VM likeliness: 20% VM confirmation: false ===== CONCLUSION: Running on bare metal ===== ``` **The corrections cost no measurable performance.** Measured against the same guest with the fidelity work abandoned entirely and every Hyper-V enlightenment enabled, and again with the patched modules against stock, the difference is smaller than the spread between runs. See [docs/RESULTS.md](docs/RESULTS.md). Run this only on hardware you control and with software you are licensed to run. --- ## How to run it ### 1. Install Three packages, all built to AUR rules and proven in a clean `extra-x86_64-build` chroot. They fetch this repo at tag `v1.1.0`, so from a checkout the same three `makepkg -si` work: ```sh git clone https://git.archworks.co/sandwich/vfio-native cd vfio-native/packaging (cd vfio-native && makepkg -si) (cd vfio-native-qemu && makepkg -si) (cd vfio-native-kvm-dkms && makepkg -si) ``` | Package | What it is | | --- | --- | | `vfio-native` | the three commands, the patches, the ACPI tables, the benchmark, the libvirt hook | | `vfio-native-qemu` | QEMU 11.1.1 with the platform-identity patches, in `/opt/qemu-native` | | `vfio-native-kvm-dkms` | the patched KVM modules, rebuilt by DKMS for every installed 7.2.x kernel | The modules land in `updates/dkms/`, which modprobe prefers, but nothing reloads them for you. With every VM shut down: ```sh sudo modprobe -r kvm_amd kvm && sudo modprobe kvm_amd ``` That gives you three commands. The post-install prints the whole flow, so you do not need to read any further to use it: | Command | | | --- | --- | | `vm-native-setup` | configure a libvirt domain | | `vm-native-verify` | measure whether it actually worked | | `vm-native-gpu` | set up GPU passthrough | Only `full` needs the QEMU and KVM packages. `native` is domain XML only. Not an Arch user? The scripts in `scripts/` are plain shell and work standalone, and the module sources build as `arch/x86/kvm` against any 7.2.x tree; `scripts/install-modules.sh` is the manual path. The packaging is convenience, not a dependency. ### 2. Fix the guest, once [docs/GUEST-SETUP.md](docs/GUEST-SETUP.md) is the full, ordered walkthrough from a plain Windows VM, including the two steps that have to happen before the disk moves to NVMe. The short version: A Windows guest that runs its own hypervisor reports that truthfully, and no host-side setting changes it - the guest really is running Hyper-V. Turning it off also makes the VM **faster**, because VBS costs 5-15% on CPU-bound workloads. ``` bcdedit /set hypervisorlaunchtype off Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All -NoRestart reg add "HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard" /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f ``` Reboot twice. Uninstall the QEMU and SPICE guest agents. Move the disk to emulated NVMe (`bus='nvme'` with a ``) - the patched QEMU will not boot a virtio disk. ### 3. Fix the host, once ```sh sudo cpupower frequency-set -g performance ``` ### 4. Configure the domain ```sh virsh -c qemu:///system shutdown win11 vm-native-setup ``` It is an interview with a default for every answer: level, cores, RAM, identity, USB, disk, Secure Boot. It reads your CPU layout itself - AMD CCDs or Intel P/E cores - sizes the guest to one cache domain, pins each vCPU onto a real SMT pair, keeps the emulator off the vCPU cores, moves the disks to emulated NVMe, replaces the virtio device set, wires Secure Boot with a key store it generates, writes the SMBIOS and ACPI identity, and gives the guest a CPU identity from the host's own generation whose thread count matches what it actually has. `-r` gives the deployment its own serials and MAC; `-u auto` passes keyboard and mouse through. Every flag answers one question in advance and `-y` takes every default. Scripted: ```sh vm-native-setup -d win11 -p full -c 8 -m 16 -r -u auto -y ``` | Flag | | | --- | --- | | `-d` | domain | | `-p` | `tuned`, `native` or `full` (default `full`) | | `-c` | guest cores; SMT doubles this into vCPUs | | `-m` | guest RAM in GiB | | `-s` | Secure Boot with enrolled keys, `on` (default) or `off` | | `-u` | USB passthrough: `none`, `auto`, or `vid:pid,...,0000:bb:dd.f` | | `-r` | randomise the hardware identity: serials, MAC, memory module | | `-y` | no prompts | It backs the domain up first and prints the revert command. Re-running it is safe. ### 5. Check it worked ```sh vm-native-verify ``` ``` OK QPC cost (ns) 14.6 OK rdtsc cost (ns) 6.6 OK 1-thread (Mops) 5254.2 OK L3 latency (ns) 10.94 OK jitter p99.99 (us) 1.600 OK stalls >100us 0 ``` `vm-native-verify` also prints whether CPUID passthrough is on, which is the last step. ### 6. CPUID passthrough, on after boot and off before the next one The `TIMER` check times the world switch on an intercepted `CPUID`, and the only way to stop paying it is to not exit. That switch has to be off while a guest cold-boots - a booting Windows enumerates CPUID bits KVM synthesises and hangs if they vanish half way through - and on once it is up. Let the libvirt hook handle that around a guest: ```sh sudo vm-native-cpuid enable win11 # off for the cold boot, on once the guest is on the network, off on stop sudo vm-native-cpuid status # what is enabled, and the live switch state sudo vm-native-cpuid disable win11 # stop automating this guest sudo vm-native-cpuid revert # remove the hook and reset the switch ``` Or drive it by hand. `vm-native-setup` prints the two lines for your declared SKU; run them once the guest is up, never before, and switch it off (`echo N`) before the next boot: ```sh echo 'AMD Ryzen 7 7700X 8-Core Processor' | sudo tee /sys/module/kvm_amd/parameters/brand_string echo Y | sudo tee /sys/module/kvm_amd/parameters/cpuid_passthrough ``` The module applies it only to vCPU threads pinned to exactly one host CPU, so on an unpinned guest it does nothing rather than something wrong. It is one switch and one brand string for the whole host, so it serves one hardened guest at a time - the hook enforces that and leaves a second guest alone. For the detection score, run VMAware in the guest **from the console session, not over SSH** - OpenSSH lands you in session 0, which is not where an interactive desktop session runs. See [docs/TESTING.md](docs/TESTING.md). --- ## The three levels All get identical performance tuning. The level only changes how much of the platform is corrected. | Level | Score | Needs | Upkeep | | --- | --- | --- | --- | | `tuned` | 13/85 | nothing | none | | `native` | 7/85 | nothing | none | | `full` | **1/85** | `vfio-native-qemu` + `vfio-native-kvm-dkms`, passthrough switched on after boot | DKMS rebuilds on kernel updates inside 7.2.x | `tuned` buys the pinning and the cache-domain sizing and nothing else. `native` is domain XML only, so it survives any host update untouched. Take it if you would rather not maintain anything. --- ## The one thing to get right **Leave the host at least 4 physical cores.** Windows calibrates the TSC at boot, and that calibration is a race. If the host cannot schedule the guest's vCPU threads cleanly through it, Windows abandons the TSC and `QueryPerformanceCounter` costs about **1300 ns instead of 15** for the rest of that boot. Software that polls the clock in a tight loop calls QPC thousands of times a second. Measured over four cold boots at each size, same XML, on a 16-core host: | Guest | Host keeps | Boots with a fast clock | | --- | --- | --- | | 24 vCPU | 4 cores | **4 / 4** | | 28 vCPU | 2 cores | 3 / 4 | | 32 vCPU | 0 cores | 2 / 4 | It is headroom, not a vCPU ceiling. 24 of 32 threads is reliable, which matters when the guest does CPU-heavy work rather than latency-sensitive work alone. `vm-native-setup` warns if you go below the margin. Because it is a race, **one measurement proves nothing**. If `vm-native-verify` reports QPC over 1000 ns, reboot and measure again before changing anything. Sizing the guest inside one cache domain is a separate trade: it takes L3 latency from about 17 ns to 10, which is worth it for latency-sensitive work and not for throughput. None of these change the outcome, measured: `useplatformclock false`, `useplatformtick no`, `disabledynamictick yes`, adding `invtsc`, pinning without leaving headroom, or which cache domain the vCPUs sit on. > **Do not put every vCPU on a realtime scheduler.** `` across a > guest sized to the whole machine booted once, failed twice, ran away to 800% CPU and left > `systemctl` unresponsive until QEMU was killed by hand. Realtime priority on as many threads as > the host has cores starves everything else, including the emulator thread the guest needs. ## The patches, by detection Every patch is grouped by the check it clears, so any of them can be taken or left. [patches/README.md](patches/README.md) is the map. **KVM** - five patches for upstream and one that is not, four detections. They correct KVM against the architecture rather than adding a layer on top of it, pass `checkpatch.pl --strict`, and ship with a cover letter and a selftest in `patches/kvm/`. | Patch | Clears | | --- | --- | | `0001-KVM-SVM-intercept-GP-when-guest-EFER.SVME-is-clear` | `SVM_EXCEPTIONS` | | `0002-KVM-x86-emulator-UD-not-GP-for-VMCALL-at-CPL-0` | `KVM_INTERCEPTION` | | `0003-KVM-x86-UD-for-KVM-hypercalls-issued-at-CPL-0` | `KVM_INTERCEPTION` | | `0004-KVM-SVM-intercept-ICEBP-and-skip-it-before-injecting-its-DB` | `DBVM` | | `0005-KVM-selftests-verify-ICEBP-DB-reports-RIP-past-the-ICEBP` | the test for 0004 | | `EXPERIMENTAL-0006-runtime-cpuid-passthrough` | `TIMER` | 0002 and 0003 are a pair - the check tries two stubs and reports whichever misbehaves first, so either alone leaves it standing. `EXPERIMENTAL-0006` is built into the DKMS package but off by default; it is not upstream material, because it hands the guest raw host CPUID. **QEMU** - eight patchsets against v11.1.1: `01-firmware`, `02-disk-identity`, `03-pci-ids`, `04-fw-cfg`, `05-usb-hid`, `06-audio`, and `07-edid` and `08-cpu-misc`, which clear no check by themselves and are carried for other detectors. --- ## GPU passthrough ```sh vm-native-gpu ``` Lists your GPUs with IOMMU groups, says which drives a display, and picks the path. **One GPU** - the normal case. The host gives the card up while the guest runs and takes it back after: ```sh sudo vm-native-gpu --single win11 0000:03:00.0 ``` Installs a libvirt hook scoped to that domain only. The host has no display while the guest runs, so get SSH working first. **Two GPUs, one spare** - easier and safer. Binds it to `vfio-pci` at boot so the host never claims it: ```sh sudo vm-native-gpu --dual 0000:0f:00.0 ``` Undo either with `sudo vm-native-gpu --revert`. --- ## When the guest is off, the host is whole No configuration here reserves host resources while the guest is not running. Static hugepages are deliberately not used, because they would take memory permanently. The CPU-isolation hook confines the host to the cores the guest is not using, and restores on every exit path - clean shutdown, hard destroy and crash, all verified. The confinement is `--runtime` only, so a reboot clears it regardless. It lives in `/etc/libvirt/hooks/qemu.d/`, which libvirt runs after any `qemu` hook you already have, so it does not replace one. --- ## Maintenance | Event | `tuned` / `native` | `full` | | --- | --- | --- | | Kernel update inside 7.2.x | nothing | DKMS rebuilds the modules; reload or reboot | | Kernel update past 7.2 | nothing | **the modules stop building** - bump `_kver` in the DKMS package | | QEMU package update | nothing | nothing, `/opt/qemu-native` is its own package | | Newer QEMU base | nothing | re-apply `patches/qemu/` | The DKMS tree carries 7.2.3's KVM sources. They build against any 7.2.x headers, and DKMS refuses anything else, so after a kernel upgrade past 7.2 the stock modules load silently, the guest boots fine and scores worse, with nothing to tell you but a pacman hook message: ```sh cat /sys/module/kvm/srcversion /sys/module/kvm_amd/srcversion modinfo -F srcversion kvm kvm_amd ``` Differ? Check `dkms status`. Check **both** modules - the SVM fixes land in `kvm-amd.ko` and the hypercall fixes in `kvm.ko`, so verifying one reports success on a stale build of the other. --- ## The last two checks **`GPU_CAPABILITIES`.** Wants a display reporting a gamma ramp, which means a real GPU passed through. Not a GPU enumeration despite the name - it is one `GetDeviceCaps` call. Out of scope for an emulated display; passing a real GPU through with `vm-native-gpu` clears it. **`TIMER` - cleared, but with a workaround.** Two detectors that OR together. The exception-latency one was first measured as a hardware floor at ratio 6.2; that was a broken probe - rebuilt to VMAware's own mechanism it reads 1.4 against a threshold of 2.5 on stock KVM, so it was never real. The instruction-latency one is the AMD world switch on an intercepted `CPUID`. `EXPERIMENTAL-0006` clears it, but as a workaround rather than a spec fix: it stops intercepting `CPUID` on a 1:1-pinned vCPU and reprograms the brand-string MSRs per core so raw `CPUID` still names the declared SKU, with RDPRU passed through because raw CPUID advertises it. Opt-in, runtime-only, and constrained to pinned vCPUs for that reason. Window 2080 -> 199 ticks, ratio 7.9 -> 0.76. `docs/ANALYSIS.md` has the whole trail, including the two wrong turns. --- ## Layout ``` patches/kvm/ five upstream-style kernel patches, a cover letter, plus one experimental patches/qemu/ eight patchsets against QEMU 11.1.1 packaging/ three AUR-shaped packages: tools, patched QEMU, DKMS modules scripts/ setup, verify, GPU, libvirt hook, module install scripts/generate-tables.py raw SMBIOS type 7/26/27/28/29 blobs for -smbios file= bench/ the benchmark and the TIMER probe, C, mingw-w64 acpi/ SSDT sources for the battery, platform devices and sensor probes docs/ANALYSIS.md why each check fires, with source references docs/RESULTS.md every measurement docs/TESTING.md how to measure it yourself ``` Longer write-up: [archworks.co/docs/vfio-native](https://archworks.co/docs/vfio-native/) --- ## Credits Written against [VMAware](https://github.com/NotRequiem/VMAware) by NotRequiem, which is the honest way to score this - it is the independent library, not a checklist that grades itself. GPL-2.0, matching the kernel and QEMU patches.