sandwich/vfio-native
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: readme, analysis, results, testing and guest setup

Browse Source
This commit is contained in:
Sandwich
2026-09-06 00:14:56 +02:00
parent 7313d04ae7
commit c1e8676834
5 changed files with 1399 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

382
README.md Normal file
View File

@@ -0,0 +1,382 @@
# vfio-native
Make a KVM guest present a hardware profile that is self-consistent with real silicon and real
firmware, and measure the result against an independent library.
A lot of software behaves differently, or refuses to run at all, once it can tell it is inside a
hypervisor. Malware samples self-disable or take a benign branch. Commercial software takes a
licensing branch. Drivers and firmware take a different code path. Any of those makes a virtual
machine useless for observing what the software would actually do on a physical machine - you end
up measuring the sandbox instead of the subject.
Two independent problems produce that gap, and this addresses both.
**KVM diverges from the AMD64 architecture** in ways a guest can read directly. The wrong
exception vector for an SVM instruction at CPL>0. `#GP` where the architecture specifies `#UD`. A
hypercall at CPL>0 that raises no exception at all. Those are spec-conformance defects, and one of
them closes a `FIXME` the kernel already carries against itself. The three KVM patches make KVM
match the architecture and nothing more, which is why they are written to be acceptable upstream.
**QEMU's emulated platform is not internally consistent** with any board that was ever
manufactured. ACPI link-device names no real firmware uses, PCI vendor and subsystem IDs that
belong to the emulator rather than to a device, firmware table fields no shipping board sets,
device identity strings that name the emulator. Correcting them makes the emulated platform look
like a platform.
Alongside both there is a VFIO tuning layer - vCPU pinning onto real SMT pairs, sizing the guest to
one cache domain, keeping the emulator thread off the vCPU cores, host CPU isolation while the
guest runs. It is independent of the fidelity work and measured separately.
Scored against [VMAware](https://github.com/NotRequiem/VMAware), which runs 85 detection
techniques. A stock KVM guest starts at 10/85. This gets it to **1/85** on both the v2.8.1 release
and current HEAD, and the one survivor needs a real GPU passed through. VMAware's own verdict:
```
VM brand: Unknown
VM likeliness: 20%
VM confirmation: false
===== CONCLUSION: Running on bare metal =====
```
**The corrections cost no measurable performance.** Measured against the same guest with the
fidelity work abandoned entirely and every Hyper-V enlightenment enabled, and again with the
patched modules against stock, the difference is smaller than the spread between runs. See
[docs/RESULTS.md](docs/RESULTS.md).
Run this only on hardware you control and with software you are licensed to run.
---
## How to run it
### 1. Install
Three packages, all built to AUR rules and proven in a clean `extra-x86_64-build` chroot. They
fetch this repo at tag `v1.1.0`, so from a checkout the same three `makepkg -si` work:
```sh
git clone https://git.archworks.co/sandwich/vfio-native
cd vfio-native/packaging
(cd vfio-native && makepkg -si)
(cd vfio-native-qemu && makepkg -si)
(cd vfio-native-kvm-dkms && makepkg -si)
```
| Package | What it is |
| --- | --- |
| `vfio-native` | the three commands, the patches, the ACPI tables, the benchmark, the libvirt hook |
| `vfio-native-qemu` | QEMU 11.1.1 with the platform-identity patches, in `/opt/qemu-native` |
| `vfio-native-kvm-dkms` | the patched KVM modules, rebuilt by DKMS for every installed 7.2.x kernel |
The modules land in `updates/dkms/`, which modprobe prefers, but nothing reloads them for you.
With every VM shut down:
```sh
sudo modprobe -r kvm_amd kvm && sudo modprobe kvm_amd
```
That gives you three commands. The post-install prints the whole flow, so you do not need to read
any further to use it:
| Command | |
| --- | --- |
| `vm-native-setup` | configure a libvirt domain |
| `vm-native-verify` | measure whether it actually worked |
| `vm-native-gpu` | set up GPU passthrough |
Only `full` needs the QEMU and KVM packages. `native` is domain XML only.
Not an Arch user? The scripts in `scripts/` are plain shell and work standalone, and the module
sources build as `arch/x86/kvm` against any 7.2.x tree; `scripts/install-modules.sh` is the manual
path. The packaging is convenience, not a dependency.
### 2. Fix the guest, once
[docs/GUEST-SETUP.md](docs/GUEST-SETUP.md) is the full, ordered walkthrough from a plain Windows
VM, including the two steps that have to happen before the disk moves to NVMe. The short version:
A Windows guest that runs its own hypervisor reports that truthfully, and no host-side setting
changes it - the guest really is running Hyper-V. Turning it off also makes the VM **faster**,
because VBS costs 5-15% on CPU-bound workloads.
```
bcdedit /set hypervisorlaunchtype off
Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All -NoRestart
reg add "HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard" /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f
```
Reboot twice. Uninstall the QEMU and SPICE guest agents. Move the disk to emulated NVMe
(`bus='nvme'` with a `<serial>`) - the patched QEMU will not boot a virtio disk.
### 3. Fix the host, once
```sh
sudo cpupower frequency-set -g performance
```
### 4. Configure the domain
```sh
virsh -c qemu:///system shutdown win11
vm-native-setup
```
It is an interview with a default for every answer: level, cores, RAM, identity, USB, disk, Secure Boot. It reads your CPU layout itself - AMD CCDs or
Intel P/E cores - sizes the guest to one cache domain, pins each vCPU onto a real SMT pair, keeps
the emulator off the vCPU cores, moves the disks to emulated NVMe, replaces the virtio device set,
wires Secure Boot with a key store it generates, writes the SMBIOS and ACPI identity, and gives
the guest a CPU identity from the host's own generation whose thread count matches what it
actually has. `-r` gives the deployment its own serials and MAC; `-u auto` passes keyboard and
mouse through.
Every flag answers one question in advance and `-y` takes every default. Scripted:
```sh
vm-native-setup -d win11 -p full -c 8 -m 16 -r -u auto -y
```
| Flag | |
| --- | --- |
| `-d` | domain |
| `-p` | `tuned`, `native` or `full` (default `full`) |
| `-c` | guest cores; SMT doubles this into vCPUs |
| `-m` | guest RAM in GiB |
| `-s` | Secure Boot with enrolled keys, `on` (default) or `off` |
| `-u` | USB passthrough: `none`, `auto`, or `vid:pid,...,0000:bb:dd.f` |
| `-r` | randomise the hardware identity: serials, MAC, memory module |
| `-y` | no prompts |
It backs the domain up first and prints the revert command. Re-running it is safe.
### 5. Check it worked
```sh
vm-native-verify
```
```
OK QPC cost (ns) 14.6
OK rdtsc cost (ns) 6.6
OK 1-thread (Mops) 5254.2
OK L3 latency (ns) 10.94
OK jitter p99.99 (us) 1.600
OK stalls >100us 0
```
`vm-native-verify` also prints whether CPUID passthrough is on, which is the last step.
### 6. Switch on CPUID passthrough, after every guest boot
The `TIMER` check times the world switch on an intercepted `CPUID`, and the only way to stop
paying it is to not exit. `vm-native-setup` prints the two lines for your declared SKU; on this
host they are:
```sh
echo 'AMD Ryzen 7 7700X 8-Core Processor' | sudo tee /sys/module/kvm_amd/parameters/brand_string
echo Y | sudo tee /sys/module/kvm_amd/parameters/cpuid_passthrough
```
Run them once the guest is up, never before: a booting Windows enumerates CPUID bits KVM
synthesises, and hangs if they vanish half way through. Switch it off again (`echo N`) before the
next boot. The module applies it only to vCPU threads pinned to exactly one host CPU, so on an
unpinned guest it does nothing rather than something wrong.
It is a module parameter, so it is one switch for the whole host: every pinned guest gets it, and
a cold boot of any of them while it is on hits the race. With more than one such guest, switch it
off before any of them boots and on again once they are all up.
For the detection score, run VMAware in the guest **from the console session, not over SSH** -
OpenSSH lands you in session 0, which is not where an interactive desktop session runs. See
[docs/TESTING.md](docs/TESTING.md).
---
## The three levels
All get identical performance tuning. The level only changes how much of the platform is
corrected.
| Level | Score | Needs | Upkeep |
| --- | --- | --- | --- |
| `tuned` | 13/85 | nothing | none |
| `native` | 7/85 | nothing | none |
| `full` | **1/85** | `vfio-native-qemu` + `vfio-native-kvm-dkms`, passthrough switched on after boot | DKMS rebuilds on kernel updates inside 7.2.x |
`tuned` buys the pinning and the cache-domain sizing and nothing else.
`native` is domain XML only, so it survives any host update untouched. Take it if you would rather
not maintain anything.
---
## The one thing to get right
**Leave the host at least 4 physical cores.**
Windows calibrates the TSC at boot, and that calibration is a race. If the host cannot schedule
the guest's vCPU threads cleanly through it, Windows abandons the TSC and
`QueryPerformanceCounter` costs about **1300 ns instead of 15** for the rest of that boot.
Software that polls the clock in a tight loop calls QPC thousands of times a second.
Measured over four cold boots at each size, same XML, on a 16-core host:
| Guest | Host keeps | Boots with a fast clock |
| --- | --- | --- |
| 24 vCPU | 4 cores | **4 / 4** |
| 28 vCPU | 2 cores | 3 / 4 |
| 32 vCPU | 0 cores | 2 / 4 |
It is headroom, not a vCPU ceiling. 24 of 32 threads is reliable, which matters when the guest
does CPU-heavy work rather than latency-sensitive work alone. `vm-native-setup` warns if you go
below the margin.
Because it is a race, **one measurement proves nothing**. If `vm-native-verify` reports QPC over
1000 ns, reboot and measure again before changing anything.
Sizing the guest inside one cache domain is a separate trade: it takes L3 latency from about
17 ns to 10, which is worth it for latency-sensitive work and not for throughput.
None of these change the outcome, measured: `useplatformclock false`, `useplatformtick no`,
`disabledynamictick yes`, adding `invtsc`, pinning without leaving headroom, or which cache
domain the vCPUs sit on.
> **Do not put every vCPU on a realtime scheduler.** `<vcpusched scheduler='fifo'>` across a
> guest sized to the whole machine booted once, failed twice, ran away to 800% CPU and left
> `systemctl` unresponsive until QEMU was killed by hand. Realtime priority on as many threads as
> the host has cores starves everything else, including the emulator thread the guest needs.
## The patches, by detection
Every patch is grouped by the check it clears, so any of them can be taken or left.
[patches/README.md](patches/README.md) is the map.
**KVM** - five patches for upstream and one that is not, four detections. They correct KVM
against the architecture rather than adding a layer on top of it, pass `checkpatch.pl --strict`,
and ship with a cover letter and a selftest in `patches/kvm/`.
| Patch | Clears |
| --- | --- |
| `0001-KVM-SVM-intercept-GP-when-guest-EFER.SVME-is-clear` | `SVM_EXCEPTIONS` |
| `0002-KVM-x86-emulator-UD-not-GP-for-VMCALL-at-CPL-0` | `KVM_INTERCEPTION` |
| `0003-KVM-x86-UD-for-KVM-hypercalls-issued-at-CPL-0` | `KVM_INTERCEPTION` |
| `0004-KVM-SVM-intercept-ICEBP-and-skip-it-before-injecting-its-DB` | `DBVM` |
| `0005-KVM-selftests-verify-ICEBP-DB-reports-RIP-past-the-ICEBP` | the test for 0004 |
| `EXPERIMENTAL-0006-runtime-cpuid-passthrough` | `TIMER` |
0002 and 0003 are a pair - the check tries two stubs and reports whichever misbehaves first, so
either alone leaves it standing. `EXPERIMENTAL-0006` is built into the DKMS package but off by
default; it is not upstream material, because it hands the guest raw host CPUID.
**QEMU** - eight patchsets against v11.1.1: `01-firmware`, `02-disk-identity`, `03-pci-ids`,
`04-fw-cfg`, `05-usb-hid`, `06-audio`, and `07-edid` and `08-cpu-misc`, which clear no check by
themselves and are carried for other detectors.
---
## GPU passthrough
```sh
vm-native-gpu
```
Lists your GPUs with IOMMU groups, says which drives a display, and picks the path.
**One GPU** - the normal case. The host gives the card up while the guest runs and takes it back
after:
```sh
sudo vm-native-gpu --single win11 0000:03:00.0
```
Installs a libvirt hook scoped to that domain only. The host has no display while the guest runs,
so get SSH working first.
**Two GPUs, one spare** - easier and safer. Binds it to `vfio-pci` at boot so the host never claims
it:
```sh
sudo vm-native-gpu --dual 0000:0f:00.0
```
Undo either with `sudo vm-native-gpu --revert`.
---
## When the guest is off, the host is whole
No configuration here reserves host resources while the guest is not running. Static hugepages are
deliberately not used, because they would take memory permanently.
The CPU-isolation hook confines the host to the cores the guest is not using, and restores on
every exit path - clean shutdown, hard destroy and crash, all verified. The confinement is
`--runtime` only, so a reboot clears it regardless. It lives in `/etc/libvirt/hooks/qemu.d/`,
which libvirt runs after any `qemu` hook you already have, so it does not replace one.
---
## Maintenance
| Event | `tuned` / `native` | `full` |
| --- | --- | --- |
| Kernel update inside 7.2.x | nothing | DKMS rebuilds the modules; reload or reboot |
| Kernel update past 7.2 | nothing | **the modules stop building** - bump `_kver` in the DKMS package |
| QEMU package update | nothing | nothing, `/opt/qemu-native` is its own package |
| Newer QEMU base | nothing | re-apply `patches/qemu/` |
The DKMS tree carries 7.2.3's KVM sources. They build against any 7.2.x headers, and DKMS refuses
anything else, so after a kernel upgrade past 7.2 the stock modules load silently, the guest boots
fine and scores worse, with nothing to tell you but a pacman hook message:
```sh
cat /sys/module/kvm/srcversion /sys/module/kvm_amd/srcversion
modinfo -F srcversion kvm kvm_amd
```
Differ? Check `dkms status`. Check **both** modules - the SVM fixes land in `kvm-amd.ko` and the
hypercall fixes in `kvm.ko`, so verifying one reports success on a stale build of the other.
---
## The last two checks
**`GPU_CAPABILITIES`.** Wants a display reporting a gamma ramp, which means a real GPU passed
through. Not a GPU enumeration despite the name - it is one `GetDeviceCaps` call. Out of scope
for an emulated display; passing a real GPU through with `vm-native-gpu` clears it.
**`TIMER` - cleared, but with a workaround.** Two detectors that OR together. The exception-latency
one was first measured as a hardware floor at ratio 6.2; that was a broken probe - rebuilt to
VMAware's own mechanism it reads 1.4 against a threshold of 2.5 on stock KVM, so it was never real.
The instruction-latency one is the AMD world switch on an intercepted `CPUID`. `EXPERIMENTAL-0006`
clears it, but as a workaround rather than a spec fix: it stops intercepting `CPUID` on a 1:1-pinned
vCPU and reprograms the brand-string MSRs per core so raw `CPUID` still names the declared SKU, with
RDPRU passed through because raw CPUID advertises it. Opt-in, runtime-only, and constrained to
pinned vCPUs for that reason. Window 2080 -> 199 ticks, ratio 7.9 -> 0.76. `docs/ANALYSIS.md` has
the whole trail, including the two wrong turns.
---
## Layout
```
patches/kvm/ five upstream-style kernel patches, a cover letter, plus one experimental
patches/qemu/ eight patchsets against QEMU 11.1.1
packaging/ three AUR-shaped packages: tools, patched QEMU, DKMS modules
scripts/ setup, verify, GPU, libvirt hook, module install
scripts/generate-tables.py raw SMBIOS type 7/26/27/28/29 blobs for -smbios file=
bench/ the benchmark and the TIMER probe, C, mingw-w64
acpi/ SSDT sources for the battery, platform devices and sensor probes
docs/ANALYSIS.md why each check fires, with source references
docs/RESULTS.md every measurement
docs/TESTING.md how to measure it yourself
```
Longer write-up: [archworks.co/docs/vfio-native](https://archworks.co/docs/vfio-native/)
---
## Credits
Written against [VMAware](https://github.com/NotRequiem/VMAware) by NotRequiem, which is the
honest way to score this - it is the independent library, not a checklist that grades itself.
GPL-2.0, matching the kernel and QEMU patches.

371
docs/ANALYSIS.md Normal file
View File

@@ -0,0 +1,371 @@
# Why each VMAware check fires, and what it took to clear it
The engineering record behind the fidelity work. `README.md` says *what to apply and when*; this
says *why*, with the source references. Written 2026-09-03 at 2/85, revised 2026-09-05 at **1/85**
on both VMAware v2.8.1 and VMAware HEAD (commit 95fecc2) - the one survivor is `GPU_CAPABILITIES`,
and VMAware's own verdict is "Running on bare metal", likeliness 20%, confirmation false.
Source read against: VMAware `src/vmaware.hpp`, QEMU 11.1.1, kernel 7.2.3 `arch/x86/kvm`.
---
## CLEARED - CPUID_SIGNATURE
**Not about hypervisor leaves.** The guest's `0x40000000` reads all zeros, and so does the
bare-metal host, so it was never the trigger.
The AMD branch (`cpuid_signature()`, vmaware.hpp:6941-6970) reads **leaf 7 subleaf 0, EDX** and
returns true if bit 26 (`IBRS`/`IBPB`), 27 (`STIBP`) or 31 (`SSBD`) is set. AMD reserves those
three to zero and enumerates its mitigations in `0x80000008.EBX` instead.
Measured with an unprivileged CPUID probe:
| | leaf 7.0 EDX | bits 26/27/31 | `0x80000008.EBX` 12/14/15/24/25 |
| --- | --- | --- | --- |
| host, bare metal | `0x10000010` | `0 0 0` | `1 1 1 1 1` |
| guest, before | `0x9c000010` | set | passed through |
| guest, after | `0x30000010` | `0 0 0` | `1 1 1 1 1` |
KVM synthesises the Intel-style `SPEC_CTRL` interface on AMD, and `host-passthrough` enables it.
Three `<feature policy='disable'>` lines fix it. All three are needed - the check ORs the bits.
Verified in-guest afterwards that every AMD-native mitigation bit survives, so the guest keeps
the interface Windows actually uses on AMD. Residual: KVM also synthesises bit 29
(`arch-capabilities`), which bare metal lacks. Nothing reads it, so it is left alone.
---
## CLEARED - SVM_EXCEPTIONS
**A bug KVM documents against itself.**
`svm.c:272` is unconditional:
```c
svm->vmcb->save.efer = efer | EFER_SVME;
```
Hardware therefore never sees the guest's real `SVME=0`. A CPL3 `VMLOAD` passes the SVME check,
hardware notices CPL is not zero, and injects `#GP` *before* the VMLOAD intercept can fire. Bare
metal raises `#UD`, and `#UD` is the one exception `svm_exceptions()` treats as innocent
(vmaware.hpp:14551).
There is a `FIXME` at `svm.c:1076-1080` describing exactly this.
The diagnostic that pinned it: the scored run printed `[ DETECTED ]` with **no debug line**.
Both noisy exits in the function call `vma_debug`, so it must have fallen through to the bare
`return true` at vmaware.hpp:14570 - which forces `svm_visible == true` and a non-`#UD` fault.
Fix, in `svm_recalc_instruction_intercepts()`:
```c
if (!(vcpu->arch.efer & EFER_SVME)) {
svm_set_intercept(svm, INTERCEPT_VMLOAD);
svm_set_intercept(svm, INTERCEPT_VMSAVE);
svm_set_intercept(svm, INTERCEPT_CLGI);
svm_set_intercept(svm, INTERCEPT_STGI);
set_exception_intercept(svm, GP_VECTOR); /* added */
} else {
```
`gp_interception()` then decodes the SVM opcode, sees `!is_guest_mode`, and calls
`svm_invoke_exit_handler(SVM_EXIT_VMLOAD)` -> `vmload_interception` ->
`nested_svm_check_permissions` (nested.c:1664), which queues `UD_VECTOR`.
The shipped `0001` also keeps the intercept across a guest clearing `SVME` later (the stock
`svm_set_efer()` dropped it there), drops it again on `SVME=1` where the erratum workaround does
not need it, and never arms it for SEV guests, whose instructions KVM cannot decode - the same
exclusion the erratum path already makes.
**Do NOT disable `svm` in guest CPUID as an alternative.** It does not change the exception, and
it moves the check into its `!svm_visible` branch, which is *worse* - "SVM hypervisor hiding CPU
capabilities", weight 150.
**Do NOT use `kvm.enable_vmware_backdoor=1`** as the no-rebuild shortcut. It arms the same
intercept, but permanently opens two well-known VMware detection surfaces in every guest on
the host: `emulate.c:2562` stops faulting CPL3 `IN`/`OUT` on ports `0x5658`/`0x5659`, and
`emulate.c:3905` makes CPL3 `RDPMC` of pseudo-counters `0x10000`-`0x10002` return host TSC,
which is architecturally impossible on real silicon. VMAware is blind to both. pafish,
al-khaser and other detection tooling are not.
---
## CLEARED - FIRMWARE
**A chain that reports only its first hit.** `firmware()` (vmaware.hpp:8860) runs six sections
in fixed order against each firmware buffer, DSDT first, and the first match returns. Fixing one
link costs a full QEMU rebuild and boot and only reveals the next - so bundle.
Links, in order:
1. ~~SMI Resources reservation string~~ - `_UID` renamed
2. ~~PRTP/PRTA routing symmetry~~ - renamed `IRQP`/`IRQA`
3. ~~Sequential PIRQ names, vmaware.hpp:9142~~ - `GSIA-H` -> `APCA-H`
4. ~~PNP0A06 resource stubs, vmaware.hpp:9148~~ - `GPER` -> `RSRA`, `PHPR` -> `RSRB`
5. ~~FACP C2/C3 latencies, vmaware.hpp:9242~~ - `0xfff` -> `0xffe`
6. Debug Port OperationRegion at `0x0402` - already dead on this tree
7. HPET register-validation loop - dormant, domain sets `hpet present='no'`. Re-arms if enabled.
8. Dummy SATA, DMAR, APIC source overrides - already dead on this tree
Link 3 requires **all four** of `LNKE`, `LNKH`, `GSIE`, `GSIH`, which is QEMU's eight-link plus
eight-GSI layout. Real boards have four PCI link devices and no GSI-named ones, so rename the
`GSI*` half and leave `LNK*` alone. Those names are declared by `build_gsi_link_dev()` and
referenced only through `build_q35_routing_table("GSI")`, which derives every name from one
3-character prefix - so a prefix change moves every reference with it.
Do not pick `IRQ` as the replacement: `_SB.PCI0.IRQA` is already the APIC routing package and a
bare NameSeg would bind to it.
`0xffe` keeps "C-state not supported" semantics (anything above 100 / 1000 means unsupported)
while not matching the check's exact-equality test against `0x0FFF`.
**WAET must stay out.** `"WAET"` is target index 18 in the section-2 scan and the table's
signature sits at offset 0, i.e. an instant hit. An earlier sed-based patcher claimed it restored
WAET and never did: `build_waet()` has zero call sites, so its `grep -q build_waet` guard was
satisfied by the definition alone and the insert never ran. Guard on a call site, not on a
definition.
---
## OPEN - GPU_CAPABILITIES (weight 20)
**Not a GPU check.** Four lines: `GetDC(nullptr)`, then `GetDeviceCaps(hdc, COLORMGMTCAPS)`,
DETECTED if the result lacks `CM_GAMMA_RAMP` (0x2) or is `CM_NONE`. No adapter enumeration, no
DXGI, no WMI, no device blacklist, no EDID - so the patched `SAM`/`SyncMaster` EDID override is
irrelevant to this check.
A plausible theory said it was a harness artefact: scoring runs over SSH, OpenSSH on Windows
lands in session 0, and a non-interactive window station has no gamma LUT. **Tested and
disproven.** Re-ran as a scheduled task with `/it`, confirmed `SessionId=1` and
`ScreenBounds={0,0,640,480}` - same score, still fires.
So it is genuine: `<video><model type='none'/>` leaves Windows on a stub display with no gamma
ramp. Real GPU passthrough fixes it. Out of scope without a spare card to pass through;
`vm-native-gpu` is the path.
An emulated adapter is not a workaround: `qxl` and `virtio-gpu` are on the DEVICES blacklist
(weight 100), and any display device can arm `BOOT_LOGO` (weight 90, brands QEMU) by giving
OVMF a GOP.
---
## CLEARED - KVM_INTERCEPTION
**The debug string is a red herring**, and it cost this project two wrong turns. It says "KVM
attempting to patch instructions on the fly", which points straight at
`KVM_X86_QUIRK_FIX_HYPERCALL_INSN`. That is not the cause.
VMAware runs its stubs at CPL3. KVM's emulator declares VMCALL in `group7_rm0[1]`
(`emulate.c:3971`) as:
```c
I(SrcNone | Priv | EmulateOnUD, em_hypercall),
```
`Priv` but **no** `PrivUD`. So `emulate.c:5348`:
```c
if ((ctxt->d & Priv) && ops->cpl(ctxt)) {
if (ctxt->d & PrivUD)
rc = emulate_ud(ctxt);
else
rc = emulate_gp(ctxt, 0);
goto done;
}
```
takes the `#GP(0)` branch, which Windows surfaces as STATUS_ACCESS_VIOLATION. The quirk code in
`emulator_fix_hypercall()` sits downstream of that CPL check and is never reached. Disabling the
quirk is a measured no-op - confirmed on three purpose-built `/dev/kvm` harnesses showing
quirk-on and quirk-off producing byte-identical `#GP` with unmodified code bytes.
The earlier "cleared INTERCEPT_VMMCALL, effect: none" log entry was therefore correct, and my
own mid-session reinterpretation of it as "closed for a bad reason" was wrong.
Both stubs need to raise `#UD`, and **either half alone leaves the detection standing** - fix
stub 0 only and the loop reaches stub 1, whose silent `-KVM_EPERM` trips the generic-hypervisor
branch instead. That is why every previous single-sided attempt measured as "no effect".
**Half A.** `PrivUD` (`emulate.c:171`, `((u64)1 << 51)`, "#UD instead of #GP on CPL > 0") added
to the VMCALL entry in `group7_rm0`. The flag had zero users in the tree before this. `Priv`
already routes to the CPL check; `PrivUD` picks the other branch. RIP does not advance.
**Half B.** `kvm_emulate_hypercall()` injects `#UD` for a CPL>0 hypercall instead of the silent
`-KVM_EPERM`. Placed **after** the Xen and Hyper-V dispatch, so enlightened guests never reach
it - and `kvm_hv_hypercall` already does the identical `cpl != 0 -> kvm_queue_exception(UD_VECTOR)`
at `hyperv.c:2546`, so this makes the KVM-PV path consistent with the Hyper-V one rather than
inventing new behaviour.
The blunt alternative - clearing `INTERCEPT_VMMCALL` - was correctly rejected: it breaks CPL0
hypercalls and bugchecks the Hyper-V domains.
Blast radius checked on the live host: `svm_patch_hypercall` writes VMMCALL and issues it at
CPL0; Linux paravirt alternative-patches to VMMCALL, also CPL0; every `KVM_HC_*` already failed
at CPL>0, so only the *shape* of the failure changes. Confirmed empirically by booting a
Hyper-V-enlightened Windows guest and a Linux paravirt guest on the patched module - both boot
and execute normally.
Half A ships as `patches/kvm/0002-KVM-x86-emulator-UD-not-GP-for-VMCALL-at-CPL-0.patch`, half B as
`patches/kvm/0003-KVM-x86-UD-for-hypercalls-issued-at-CPL-0.patch`. Both land in `kvm.ko`, not
`kvm-amd.ko`, so verify `/sys/module/kvm/srcversion` - checking only `kvm_amd` reports success on
a stale build.
---
## CLEARED - TIMER
**There is no clock to lie to.** The check does not use the TSC. It times a bare `CPUID` (EAX=0)
against eight `_mm_lfence()`, using a second thread on another core spinning `counter++` on a
64-byte-aligned volatile; the "tick" is a cross-core cache-line bounce (the code says so at
vmaware.hpp:7480, "this is a cache-based counter").
`serialize_available = cpu::is_intel()` is hard-false on AMD, so the LFENCE branch always runs.
Threads are pinned *inside* the guest with `SetThreadGroupAffinity`. The estimator takes a
500-1000 sample batch per trial, an interquartile mean of the middle 50%, then the **minimum
across 5 trials**, with samples bracketed by counter-edge spin-waits and followed by
`burn_random_cycles()` so a hypervisor cannot predict when to freeze the counter thread.
**There are TWO detectors, and they OR together** (vmaware.hpp:7484, 7488, 7493). Clearing one
achieves nothing.
### Detector 2 was never the wall - the probe was wrong
The first version of `bench/timerprobe.c` reported the exception-latency detector at **6.169**
against a threshold of 2.5, and two days of work treated that as a hypervisor floor. It was a
measurement error. VMAware's software side is `RtlCaptureContext` + `ZwRaiseException` inside an
SEH frame with `EXCEPTION_EXECUTE_HANDLER`, which is a syscall in, a kernel dispatch, and an
`RtlUnwindEx` out. The probe used `RaiseException` under a vectored handler, which never enters the
kernel on the software side, so its reference window was several times too short and the ratio
several times too high.
Rebuilt mechanism for mechanism (`__C_specific_handler` scope table, `ZwRaiseException`,
`EXCEPTION_EXECUTE_HANDLER`), the probe reads **1.47**, and VMAware's own debug line agrees on
the same boot with the stock `#DB` intercept in place:
TIMER: Exception > VMM -> 10167 | nVMM -> 7066 | Ratio -> 1.439
Detector 2 passes on stock KVM. No bare-metal Windows baseline was taken - the brief asked for
one, no physical Windows machine was available - so the only claim made here is the measured one:
the guest is under the threshold with `#DB` intercepted, by a margin of 1.1.
### Detector 1 is a world switch, and the only way to not pay it is to not exit
Measured: `cpuid 2054-2383 / lfence 262-310 ticks`, ratio 7.6-8.2, threshold 2.5. The exit itself
is ~1800 ticks, on the order of 400 ns, an ordinary Zen 4 world switch. Cross-CCD counter
placement, the exit fastpath, TSC offsetting and CPUID-leaf overrides were all measured and all
failed (table below).
`EXPERIMENTAL-0006-runtime-cpuid-passthrough` clears `INTERCEPT_CPUID` on every guest entry once
`cpuid_passthrough=1`, so `CPUID` runs on the silicon. Three things follow from raw CPUID, and the
patch handles each:
- **The brand string reverts to the host SKU.** The AMD Processor Name String MSRs
(`0xC0010030-35`) back `CPUID 0x80000002-4` directly and are writable and per-thread on Zen 4,
so the declared SKU is written on the pinned core at entry and restored in `svm_vcpu_put()`.
`THREAD_MISMATCH` stays clear; VMAware's own thread database resolves `7700X` to 16 threads.
- **Raw CPUID advertises RDPRU** (`0x80000008 EBX[4]`), which KVM masks out of its own CPUID and
intercepts with `kvm_handle_invalid_op()`, i.e. `#UD`. VMAware's `INTERRUPT_SHADOW` and
`SINGLE_STEP` both have an RDPRU variant gated on exactly that bit, and both report the `#UD` as
an "exception anomaly, hypervisor seems to be present with CPUID interception disabled". Two
full scans fired both checks before this was understood, and an exact-stub reproducer with a
`kvm_exit` trace showed the TF `#DB` after a native CPUID landing at the right RIP every time -
the single-step path was never at fault. The RDPRU intercept is cleared together with the CPUID
one; the guest then reads the pinned core's real `MPERF`/`APERF`.
- **Raw CPUID must only be enabled after boot.** Windows enumerates KVM's synthetic leaf-1 bits
(`x2apic`, `tsc-deadline`) during boot and hangs if they vanish mid-enumeration. The module
parameter is runtime-toggled for this reason.
The intercept is also withheld from any vCPU thread whose allowed-CPU mask is wider than one CPU
(`cpumask_weight(current->cpus_ptr) == 1`), so the brand override cannot leak onto a host core: the
patch verifies the 1:1 pinning itself rather than trusting the operator.
Measured with the passthrough on, VMAware HEAD debug, console session:
TIMER: Instruction > VMM -> 219 | nVMM -> 290 | Ratio -> 0.755
TIMER: Exception > VMM -> 10147 | nVMM -> 7551 | Ratio -> 1.344
VM detections: 1/85 ===== CONCLUSION: Running on bare metal =====
The `#DB` intercept is left in place. Clearing it too was measured (detector 2 drops to 0.96) and
gains nothing the score needs, while it removes the single-step re-injection KVM relies on for
its own NMI-window logic. An earlier note here blamed that clear for the `INTERRUPT_SHADOW` and
`SINGLE_STEP` detections; that was wrong, RDPRU was the cause both times.
Measured and dead before the passthrough:
| Attempt | Result |
| --- | --- |
| BetterTiming TSC compensation | no detection change, 6x slower boot (6.1 s -> 38.6 s CPU) |
| Hypervisor-Phantom CPUID override | ratio halved to ~810 ticks, **still detected**, and added `SINGLE_STEP` (10 -> 11) |
| Clear `INTERCEPT_CPUID` from boot | guest never boots |
| All four intercepts cleared | triple fault, `EFER=0`, instant |
| CPUID exit fastpath in `svm_exit_handlers_fastpath` | the fastpath runs *after* the world switch (`svm.c:4429`, called at `:4664`), so it cannot remove the exit that is being measured |
| Cross-CCD topology to grow the baseline | the measured ceiling falls ~1.7x short, and VMAware picks its own nearby core pair anyway |
| Both of the above combined | still short |
---
## CLEARED - DBVM (weight 150, VMAware HEAD only)
Added to VMAware after v2.8.1; the release build does not fire it, HEAD does, with
`DBVM: ICEBP failed to advance guest RIP`. Named after a hypervisor-backed debugger, but the sub-check
that fires is generic: it clears DR0-DR7 through `NtSetContextThread`, executes `F1 C3`
(`icebp; ret`) under SEH, and compares the exception context's RIP against `stub + 1`.
`ICEBP` (`INT1`) raises a trap-like `#DB`, so the frame RIP on hardware is the next instruction.
The first theory here - that SVM's `#DB` exception intercept catches it and `db_interception()`
re-injects it with RIP still on the `F1` byte - was wrong, and it was wrong in a way that only a
trace could show: on Zen 4 the exception intercept never fires for ICEBP at all, and every warm
reproducer read the correct `+1`. What the `kvm_exit` trace of a cold selftest shows instead:
kvm_exit: reason npf rip 0x402ec4 ... intr_info 0x80000301 <- ICEBP at 0x402ec4
kvm_inj_exception: #DB [reinjected]
The `#DB` was being delivered when it hit a nested page fault on an unmapped IDT/handler page.
The exit carries the pending `#DB` in `EXITINTINFO` with the saved RIP on the ICEBP, and
`svm_complete_interrupts()` re-queues it as a plain hardware exception, which the next `VMRUN`
injects at that RIP. So the guest sees the ICEBP's own address whenever the delivery touches a
page KVM has not mapped yet - deterministic in a fresh VM, sporadic in a running Windows guest
(page compaction and reclaim keep unmapping NPT entries), which is why the scanner hit it and the
warm probes did not.
`0004-KVM-SVM-intercept-ICEBP-and-skip-it-before-injecting-its-DB` enables SVM's dedicated
`INTERCEPT_ICEBP`, which fires before the `#DB` exists, skips the instruction with
`svm_skip_emulated_instruction()` and queues the `#DB` with the same DR6 payload
`db_interception()` uses. Once RIP has been advanced, an injection that is itself interrupted is
re-injected with the advanced RIP. This is the shape VMX already has (`is_icebp()` in
`vmx/vmcs.h`, skipped in `handle_exception_nmi()`).
Proof, at CPL0 with no Windows involved - a KVM selftest (`x86/icebp_test.c`, shipped as 5/5 of
the series) executes `icebp` under a `#DB` handler that records the frame RIP:
unpatched svm.c: 0x402ec4 != 0x402ec5 (db_rip != next_rip) FAIL
with 0004: PASS
and the two full scans with 0004 in place no longer fire `DBVM`.
---
## Method notes worth keeping
**Score from the console session.** OpenSSH on Windows puts you in session 0, the services
session. Every score this project took for its first two days came from there. Use a scheduled
task with `/it` and confirm `(Get-Process -Id $PID).SessionId` is 1 - `schtasks /run` reports
success regardless of where the task actually ran.
**One change per boot.** The 70-second silent power-off from an ACPI namespace failure has no
error message and no log line. The only way to attribute it is to have changed one thing.
**Read the debug line, not the count.** `[ DETECTED ]` with no debug line is not "cleared" -
some targets return silently (`Xen` at vmaware.hpp:9174, `BXPC` at :9186).
---
## Status
VMAware v2.8.1 debug: 1/85 GPU_CAPABILITIES Running on bare metal
VMAware HEAD debug: 1/85 GPU_CAPABILITIES Running on bare metal, likeliness 20%
with `0001-0004` in the modules, `EXPERIMENTAL-0006` built in and enabled after boot on a 16 vCPU
guest pinned 1:1 to one CCD, declaring a Ryzen 7 7700X. The remaining check needs a real GPU passed
through, which needs a spare card. `bench/timerprobe.c` reproduces both `TIMER` detectors and
`bench/`-adjacent probes for the others live in `docs/TESTING.md`.

198
docs/GUEST-SETUP.md Normal file
View File

@@ -0,0 +1,198 @@
# From a plain Windows VM to the tuned, corrected one
`vm-native-setup` does the host side in one pass. This is the guest side, in the order that
works, plus the two host steps that have to happen between them. Everything here was run on a
Windows 11 guest; the order is the part that cost time to learn.
The shape of it:
```
base domain guest host guest again
----------- ----- ---- -----------
virtio disk, QXL, -> 1. hypervisor + VBS off -> 4. vm-native-setup -> 5. remove virtio
agents, 52:54:00 MAC 2. SSH + RDP on (disk to NVMe, devices, drivers + agents
3. stornvme boot-start firmware, identity) -> 6. score, verify
```
Steps 1 to 3 are done with the guest still on its virtio disk. Step 4 changes the disk bus, and
step 5 is only safe after that: remove `viostor` while the disk is still virtio and the next boot
is `INACCESSIBLE_BOOT_DEVICE`.
---
## 1. Turn off the guest's own hypervisor
A Windows guest that runs Hyper-V reports that truthfully, and no host-side setting changes it.
Virtualization-based security also costs 5 to 15 percent on CPU-bound work, so this is the one
step that makes the VM faster as well as quieter. As Administrator, in order:
```
bcdedit /set hypervisorlaunchtype off
bcdedit /set vsmlaunchtype off
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard /v RequirePlatformSecurityFeatures /t REG_DWORD /d 0 /f
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\HypervisorEnforcedCodeIntegrity /v Enabled /t REG_DWORD /d 0 /f
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\CredentialGuard /v Enabled /t REG_DWORD /d 0 /f
reg add HKLM\SYSTEM\CurrentControlSet\Control\Lsa /v LsaCfgFlags /t REG_DWORD /d 0 /f
reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f
reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard /v HypervisorEnforcedCodeIntegrity /t REG_DWORD /d 0 /f
```
Then the optional features that bring a hypervisor with them. WSL is one of them and it is lost:
```powershell
Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All, Microsoft-Windows-Subsystem-Linux, VirtualMachinePlatform, HypervisorPlatform, Containers-DisposableClientVM -NoRestart
```
Reboot twice. Verify:
```powershell
(Get-CimInstance Win32_ComputerSystem).HypervisorPresent # False
(Get-CimInstance -Namespace root\Microsoft\Windows\DeviceGuard Win32_DeviceGuard).VirtualizationBasedSecurityStatus # 0
```
**Some installs will not let go.** On one Windows 11 install here the status stayed at 2 through
nine reboots with every one of the settings above applied. Memory Integrity had been on since
install and the policy is enforced from a place none of these keys reach. The two things that
worked: a clean install where VBS was never enabled, or the Core Isolation > Memory Integrity
toggle in Windows Security, then the commands again. Check the two values before anything else,
because every later step assumes they read `False` and `0`.
## 2. Two ways in that survive a dark console
The full profile removes the emulated display, and GPU passthrough hands the host's screen to the
guest. Both need a way in that does not depend on a console. As Administrator:
```powershell
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
Set-Service sshd -StartupType Automatic; Start-Service sshd
New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server" /v fDenyTSConnections /t REG_DWORD /d 0 /f
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"
```
Put your public key in `C:\ProgramData\ssh\administrators_authorized_keys` for an Administrator
account. `vm-native-verify` uses that key.
SSH into the guest fails with `Corrupted MAC on input` until the host has the e1000e offload rule
from the `vfio-native` package. The emulated NIC's TX offloads corrupt integrity-checked traffic on
the host side of the tap; SMB tolerates it, SSH does not. The package installs a udev rule that
turns the offloads off on every libvirt tap as it appears, and `vm-native-setup` says so if it is
missing.
## 3. Make the NVMe driver boot-critical
The disk is about to move from virtio to emulated NVMe, and Windows only loads boot-start drivers
before it can read the disk. `stornvme` is inbox but not always boot-start on an install that
never saw an NVMe disk. Before the move:
```
reg add HKLM\SYSTEM\CurrentControlSet\Services\stornvme /v Start /t REG_DWORD /d 0 /f
reg add HKLM\SYSTEM\CurrentControlSet\Services\storahci /v Start /t REG_DWORD /d 0 /f
reg delete HKLM\SYSTEM\CurrentControlSet\Services\stornvme\StartOverride /f
reg delete HKLM\SYSTEM\CurrentControlSet\Services\storahci\StartOverride /f
```
The `reg delete` lines may say the key does not exist, which is fine. Skip this step and the
first NVMe boot ends at Windows Boot Manager with `0xc0000225`, "a required device isn't connected
or can't be accessed". That is exactly what a set-aside copy of the daily VM's disk did here.
Shut the guest down.
## 4. The host pass
```sh
vm-native-setup -d win11 -p full -r
```
It moves every disk to emulated NVMe with a serial, replaces the virtio device set (balloon,
RNG, serial channel, agent channel, virtiofs, virtio NIC and inputs) with what a real board has,
removes the emulated display for the full level, wires Secure Boot with a key store it generates,
writes the SMBIOS and ACPI identity, pins the vCPUs, and points the domain at the patched QEMU.
`-r` gives this deployment its own serials, MAC and memory module, which matters because every
installation of a tool that ships fixed identity strings shares one fingerprint. The generated
values live in `~/.local/share/vfio-native/<domain>/identity.env` and stay put until you pass `-r`
again.
Two things it asks or warns about:
- **USB passthrough.** It lists your devices and which controller each sits behind. `auto` passes
keyboard and mouse: a whole controller when only they sit on it and its IOMMU group is clean,
the individual devices otherwise. The host loses whatever is passed for as long as the guest
runs, so have SSH working first.
- **The firmware store.** Enrolling Secure Boot keys means the domain's EFI variable store is
recreated from the new template. Boot entries come back on their own. If BitLocker is on in the
guest, suspend it first or the next boot asks for the recovery key.
Before the first boot, if the host has less free memory than the guest's RAM, free and compact
it so the guest lands on transparent hugepages; `vm-native-setup` prints the two commands when it
applies. The NIC stays `e1000e`, so the network survives the driver removal in the next step. Do not use
virtiofs for host files: it is a virtio device the scanner names, and its shared memory backing
blocks transparent hugepages for the whole guest. Share over SMB on the e1000e link instead.
## 5. Remove the virtio drivers and the agents
Boot the guest. It is on NVMe now, so `viostor` and `vioscsi` can go. In this order, as
Administrator:
1. Uninstall `virtio-win-guest-tools` from Apps, or its cached installer with `/uninstall /quiet`.
Then `msiexec /x` whatever it leaves behind: the QEMU guest agent, the Spice agent, the Red Hat
QXL driver, the virtio-win driver installer.
2. Remove the Red Hat packages from the driver store:
```
pnputil /enum-drivers
pnputil /delete-driver oemNN.inf /uninstall /force
```
one per `Red Hat` or `virtio` entry the first command lists.
3. Delete the service keys they leave behind:
```
for %s in (viostor vioscsi netkvm vioser BalloonService VirtioFsSvc pvpanic vioinput viorng qemu-ga spice-agent) do reg delete HKLM\SYSTEM\CurrentControlSet\Services\%s /f
```
Reboot. Nothing in the guest now names the emulator.
## 6. Score and verify
From the host, with the guest up for a minute:
```sh
echo 'AMD Ryzen 7 7700X 8-Core Processor' | sudo tee /sys/module/kvm_amd/parameters/brand_string
echo Y | sudo tee /sys/module/kvm_amd/parameters/cpuid_passthrough
vm-native-verify
```
`vm-native-setup` printed those two lines with your SKU. The switch is host-wide: it applies to
every pinned guest at once, and a cold boot of any of them while it is on hits the enumeration
race, so with several guests turn it off before any boots and on when they are all up. Then the
scanner, from the console session, as `docs/TESTING.md` describes. The reference guest reads 1/85.
## 7. GPU and the rest of the desk
```sh
sudo vm-native-gpu --single win11 0000:03:00.0 /path/to/vbios.rom
```
writes every function of the card into the domain, adds the vBIOS as the ROM file if you give
one, removes the emulated display, and installs the hook that frees the card when the guest starts
and gives it back when it stops. The host has no screen while the guest runs. `--dual` is the same
without the hook, for a second card the host does not use. USB devices were handled in step 4;
re-run `vm-native-setup -u <spec>` to change them.
---
## What is not fixed by any of this
- **`GPU_CAPABILITIES`** needs the real GPU. Nothing emulated passes it.
- **A guest that has run its own hypervisor.** If step 1's two values will not read `False` and
`0`, no host setting rescues it. Reinstall.
- **Windows activation.** `-r` changes the board serial and the MAC. A digital licence tied to
the previous hardware hash may want re-activation. Run `-r` once, early, not per boot.
- **The EDID.** The emulated monitor's serial is compiled into the `07-edid` QEMU patch and `-r`
does not touch it. It only exists while the guest still has an emulated display; the full level
has none.
- **Memory module serials.** QEMU takes one string set for every DIMM, so all modules report the
same serial. Real boards do not, and nothing scored here reads it.

154
docs/RESULTS.md Normal file
View File

@@ -0,0 +1,154 @@
# Performance: what the corrected guest costs
Benchmarks are `bench/vmbench.c`, cross-compiled with mingw-w64 and copied into the guest.
Nothing is installed in the guest. No GPU is passed through, so every number here is a CPU,
memory, scheduling or clock proxy for interactive and CPU-bound workloads - there is no graphics
figure to be had.
Run `scripts/verify-perf.sh` to reproduce any of this.
## The corrections are free
**Ceiling** is the same guest with the fidelity work abandoned entirely - hypervisor visible, the
full Hyper-V enlightenment set, `hypervclock`. It is not a usable config, only a reference for how
fast this guest can possibly go.
**Conformant** is the shipping config, the `full` level: 16 vCPU pinned to one CCD, hypervisor
CPUID bit cleared, no enlightenments, patched QEMU, patched KVM modules, declaring a Ryzen 7
7700X. Scores 1/85, verdict "Running on bare metal".
Medians of 3 sequential runs each, patched modules in both, host governor `performance`:
| | Conformant | Ceiling (detectable) | Difference |
| --- | --- | --- | --- |
| int latency, Mops | 880 | 882 | none |
| int throughput, Mops | 5258 | 5294 | -0.7% |
| fp, Mops | 3512 | 3566 | -1.5% |
| QPC cost, ns | 15.1 | 14.5 | none |
| jitter p99.99, us | 1.7 | 1.4 | none |
| stalls >100us / 10 s | 0-2 | 0-1 | none |
Run-to-run spread inside each config (874-894 Mops conformant, 874-893 ceiling) is wider than the
gap between them. **The full correction set costs nothing measurable.** There is no performance
argument for leaving the guest uncorrected.
The Hyper-V enlightenments in particular buy nothing here. The one place they were expected to
matter is the clock, and once the guest is sized to one CCD Windows uses the TSC for QPC by
itself, leaving `hypervclock` nothing to improve.
### The KVM patches are free too, and so is the CPUID passthrough
Measured 2026-09-05 on one boot each, same XML, three sequential `vmbench all` runs per row, the
guest settled 60 s, nothing else running in it, host governor `performance`, the isolation hook
confining the host to the twelve threads the guest does not use. Three module builds:
| | `0001-0003` | `0001-0006`, passthrough off | `0001-0006`, passthrough on |
| --- | --- | --- | --- |
| int throughput, Mops | 5308 / 5315 / 5318 | 5315 / 5315 / 5311 | 5256 / 5246 / 5250 |
| QPC cost, ns | 14.4 / 14.8 / 14.6 | 14.4 / 14.4 / 14.5 | 14.4 / 14.7 / 14.7 |
| L3 latency, ns | 10.26 / 10.19 / 10.20 | 10.21 / 10.29 / 10.35 | 10.40 / 10.39 / 10.39 |
| jitter p99.99, us | 1.4 / 1.3 / 1.3 | 1.4 / 1.4 / 1.4 | 1.4 / 1.4 / 1.4 |
| stalls >100us / 10 s | 0 | 1 / 0 / 0 | 0 / 0 / 0 |
| DRAM read, GB/s | - | 52.4 / 52.2 / 51.6 | 49.5 / 51.5 / 52.1 |
The passthrough-on column reads 1.2% lower on integer throughput, and every one of its three runs
sits below every off run. That is exactly the shape a real cost would have, so it was re-run
interleaved, off then on, three times, `vmbench cpu` only:
off 5285.5 on 5276.0
off 5285.5 on 5329.0
off 5329.3 on 5330.5
Gone. The first batch's gap was an order effect: it ran off-then-on straight after a QEMU build on
the host, while the host load was still decaying. **Benchmarks that share a machine with other
work measure the sharing, even when the other work has just finished.** Interleave, or the
ordering becomes the result.
What the passthrough actually costs, measured from the host side:
- **vcpu_put rate**: ~2300 context switches per second across the 16 vCPU threads, ~143 per vCPU,
idle or under a one-thread load. Each one restores and re-applies the six brand-string MSRs.
- **Brand MSR writes**: six `WRMSR` through `/dev/cpu/8/msr` take 791 ns against 712 ns for six
`RDMSR` the same way, so at most ~130 ns per write including the syscall. Twelve writes per
put/load cycle at 143 per second is under 0.03% of a vCPU.
- **Unpinned guest**: the module withholds the passthrough from any vCPU thread whose allowed-CPU
mask is wider than one CPU, checked on every entry, so the writes never happen. Measured with
`virsh vcpupin 0 0-31 --live` on a running guest: the CPUID window goes from 199 ticks back to
2440, and returns to 199 when re-pinned.
### A correction worth recording
An earlier version of this file reported a 3% cost for the corrections. That was wrong, and it was
a measurement bug rather than a real effect: the repeat runs launched `clock`, `cpu` and `jitter`
with `&`, so all three ran concurrently and competed for the same cores. Running them
sequentially removes the gap. Benchmarks that share a machine with other benchmarks measure the
sharing.
## The finding that mattered
**Windows' boot-time TSC calibration is a race, and losing it costs 85x on every timing call.**
QPC then costs ~1300 ns instead of ~15 for the rest of that boot. `rdtsc` stays cheap either way,
so the TSC itself is fine; it is Windows' decision that changes.
What decides it is how much of the host the guest leaves alone. Four cold boots at each size,
same XML:
| Guest | Host keeps | Fast clock |
| --- | --- | --- |
| 24 vCPU | 4 cores | 4 / 4 |
| 28 vCPU | 2 cores | 3 / 4 |
| 32 vCPU | 0 cores | 2 / 4 |
An earlier version of this file reported a hard 16-vCPU ceiling. That was wrong twice over: it
came from two data points with nothing tested between them, and one sample at each, when the
behaviour is probabilistic rather than a threshold.
Ruled out by measurement, not assumption: the clock-source BCD settings
(`useplatformclock false`, `useplatformtick no`, `disabledynamictick yes`) change nothing;
`invtsc` changes nothing; pinning alone changes nothing; the cache-domain span changes nothing.
Realtime vCPU scheduling (`<vcpusched scheduler='fifo'>`) makes it actively worse: on a guest
sized to the whole machine it failed to boot on two of three attempts and wedged the host.
The second-order effect is larger than the clock tax. Going from 32 to 16 vCPU takes stalls over
100 us from 452 per 10 s to 0-2, and p99.99 from 87 us to 1.6 us. That tail is what shows up as
hitching in an interactive session, and mean throughput does not show it.
Dropping to 16 vCPU also gains ~5% single-thread and takes L3 latency from 17.4 to 10.2 ns,
because 8 cores on one CCD share one L3 rather than straddling the Infinity Fabric.
## Why the guest declares a Ryzen 7 7700X
16 vCPU contradicts a Ryzen 9 7950X, which is 32 threads, and VMAware's thread-count check reads
the **CPUID** brand string - not SMBIOS, so the injected SMBIOS type 4 string is irrelevant to it.
The guest therefore declares a Ryzen 7 7700X, a real 8-core/16-thread Zen 4 part. The thread count
is then correct and the check passes. It also makes the guest internally consistent: it genuinely
is an 8-core Zen 4 chip on a single CCD, which is exactly what a 7700X is.
That is the thesis of the whole project in one example. The check does not stop firing because a
better identity string was picked. It stops firing because the declared identity finally agrees
with the silicon underneath it. Parts of a machine that contradict each other are exactly what a
detection technique looks for.
## Known limits of this measurement
- `c2c` is not trustworthy and no conclusion is drawn from it. It reports ~34 ns for what should
be an SMT pair, and moved 38 -> 127 ns between near-identical configs. The
`InterlockedExchange` ping-pong's own overhead likely dominates.
- `cpu_mt` at 8 threads is a harness artefact: it pins to CPUs 0-7, which in an 8c/2t guest are
four cores' SMT pairs rather than eight cores, so it measures SMT contention, not scaling.
- A background compute workload on the host is the most likely source of the occasional
multi-millisecond outlier in the jitter tail, and it is not controlled for.
- Hugepage coverage is a host-state figure, not a config figure: the same guest read 16% and 49%
of resident memory on 2 MiB pages on two boots, depending on free memory and compaction before
the start (`docs/TESTING.md`). None of the numbers above were taken with the coverage checked,
which is one reason to trust the interleaved comparisons more than absolute values.
- The `diskio` rows are not used for any conclusion: `rand4k_IOPS` moved between 5139 and 13650
across otherwise identical runs, which is the host page cache, not the guest.
- The `vm-native-verify` sample quoted in the README and the docs page read 5012 Mops single-thread,
4% under the 5220-5330 the controlled runs above read on the same build. It was one run, taken
a minute after boot with the guest still settling. `vm-native-verify` is a pass/fail gate, and
its threshold is set for that; the cost claims here rest only on the interleaved runs.
- The `#GP` intercept's individual cost has not been isolated; the three-column table above
bounds the whole five-patch set at "within run-to-run spread".

294
docs/TESTING.md Normal file
View File

@@ -0,0 +1,294 @@
# Testing methodology
Two separate questions, measured separately: **does the platform read as native**, and **is it
fast**. A configuration can pass one and fail the other, and most published tuning advice is never
measured at all.
---
## Measuring the detection score
### The scanner
[VMAware](https://github.com/NotRequiem/VMAware) runs 85 techniques and reports which fired. Use
the debug build, which prints *why* each one fired - that string is the only efficient way to work
through them.
```
vmaware_debug.exe --all --no-ansi
```
Grading with this project's own checklist instead is generous: it tests what the project already
fixed. VMAware is independent of it, which is the point.
Score against **two** builds: the latest release (`vmaware_debug.exe` from the GitHub release page,
v2.8.1 at the time of writing) and a debug build of HEAD. HEAD grows checks between releases -
`DBVM` arrived after v2.8.1 and fired on this guest while the release build said 2/85. HEAD builds
for Windows with clang (`--target=x86_64-w64-mingw32` against the mingw sysroot) with `-DVMAWARE_DEBUG`;
one file with a frameless SEH leaf needs the gcc assembler, so build that object with
`x86_64-w64-mingw32-gcc` and link the rest with clang.
### Score from the console session, not over SSH
OpenSSH on Windows drops you in **session 0**, the services session. That is not where an
interactive application runs, and it has no real display. Run it as a scheduled task instead:
```
schtasks /create /tn VMAware /tr "C:\path\run.cmd" /sc once /st 00:00 /it /rl highest /f
schtasks /run /tn VMAware
```
`schtasks /run` reports success regardless of where the task actually ran, so confirm from inside
it:
```powershell
(Get-Process -Id $PID).SessionId # must be 1, not 0
```
Run as Administrator either way - several checks need it.
This one cost real time here. A plausible theory said `GPU_CAPABILITIES` was a session-0 artefact,
because a non-interactive window station has no gamma LUT. Re-running it confirmed in session 1
gave the identical score with the check still firing. It was genuine, and the theory was
comfortable rather than correct.
### Read the debug line, not the count
`[ DETECTED ]` with no debug line above it does **not** mean cleared - some checks return
silently. Conversely, a check firing with an unexpected debug string usually means your model of
it is wrong.
`SVM_EXCEPTIONS` was diagnosed entirely from the *absence* of a debug line: both noisy exits in
that function log something, so silence forced the conclusion that it fell through to the bare
`return true`, which in turn pinned down exactly what the exception must have been.
### The guest ignores ACPI shutdown when it feels like it
`virsh shutdown` is a request, and a Windows guest with a dialog open or an update pending sits
there. Every script here waits, then asks from inside:
```sh
ssh User@guest 'shutdown /s /t 0 /f'
```
and only then touches modules. `modprobe -r kvm` with the domain still up fails, and a module
swap racing a guest that is still shutting down is how a scan ends up measuring the wrong build.
### One change per boot
The ACPI failure mode is a guest that powers itself off after about 70 seconds having read nothing
from disk. No error, no log line, nothing in the journal. The only way to attribute it is to have
changed exactly one thing since the last known-good boot.
Snapshot before any firmware change:
```sh
qemu-img snapshot -c pre-acpi-$(date +%Y%m%d) /path/to/guest.qcow2
```
### Chains report only their first hit
`FIRMWARE` walks about a dozen fingerprints and returns on the first match, so fixing one link
costs a full rebuild and boot and only reveals the next. Bundle every edit into one rebuild, or you
will spend an evening discovering links one at a time.
---
## Measuring performance
### Why not an application benchmark
The benchmark guest has no GPU, so there is no graphics figure to measure. Everything here is a
CPU, memory, scheduling or clock proxy. That is a real limitation and worth stating plainly rather
than implying the numbers cover a whole application.
They are chosen because each one stands in for something an interactive or CPU-bound workload
actually does:
| Measurement | What it stands in for |
| --- | --- |
| QPC cost | software that polls the clock in a tight loop calls QPC thousands of times a second |
| single-thread throughput | the main thread of a latency-sensitive process |
| memory latency | the dominant cost in most pointer-chasing work |
| core-to-core latency | work-queue handoffs between threads |
| jitter tail | hitching in an interactive session |
| storage IO | bulk load and streaming of data off disk |
### The harness
`bench/vmbench.c`, cross-compiled with mingw-w64 and copied in. **Nothing is installed in the
guest** - no Cinebench, no AIDA64, nothing that would itself be a detectable artefact.
```sh
x86_64-w64-mingw32-gcc -O2 -o vmbench.exe vmbench.c
```
`vm-native-verify` does the build, copy, run and grading for you.
### How the harness avoids lying
- **A volatile sink** on every loop result, so the optimiser cannot delete the work.
- **Pointer-chase over a random single cycle** for memory latency, one node per 64-byte line. A
strided walk would be prefetched and would measure bandwidth instead.
- **`QueryPerformanceCounter` for timing, never `rdtsc`.** TSC behaviour is one of the things this
project changes, so timing with it would measure the instrument.
- **Percentiles, not means,** for jitter. The tail is the whole point.
- **Warm-up passes** before every timed section.
- **Minimum across repetitions** where the metric is a floor, interquartile mean where it is a
distribution.
### The mistakes that produce fake results
**Measure a race more than once.** Windows' boot-time TSC calibration either succeeds or does not,
and the result holds for that whole boot. The same XML gives 15 ns on one boot and 1250 ns on the
next when host headroom is tight. A single sample per configuration produced a confident,
published, wrong conclusion here - a hard "never exceed 16 vCPUs" rule drawn from two data points
with nothing tested between them. Four boots per configuration is the minimum for anything
boot-dependent.
**Never run benchmarks concurrently.** This produced a phantom "3% cost of the corrections" here
that survived into a written conclusion before it was caught. The repeat loop was:
```sh
vmbench clock & vmbench cpu & vmbench jitter 0 10
```
Three benchmarks competing for the same cores. Both configurations were measured the same way, so
the comparison *looked* controlled, and the artefact still leaked through because the two schedule
contention differently. Sequential runs removed the gap entirely.
**Benchmark a settled guest.** A machine that just booted is indexing, patching and starting
services, and all of it lands in the stall counts. `vm-native-verify` waits 60 seconds by default.
**Set the host governor first.** `powersave` costs 2-3% and makes everything noisier.
**Watch for host confounders.** A background compute workload on the host is the most likely
source of the occasional multi-millisecond outlier in the jitter tail. It is not controlled for,
and that is stated rather than left out.
### Expected numbers
On a 7950X with the guest correctly configured:
| Reading | Expected | If wrong |
| --- | --- | --- |
| QPC cost | ~15 ns | over 1000 ns means the boot TSC calibration lost its race; reboot and re-measure |
| rdtsc cost | ~7 ns | - |
| 1 thread | ~5300 Mops | host governor is not `performance` |
| L1 / L2 / L3 | ~0.8 / ~3.4 / ~10 ns | high L3 means vCPUs across two cache domains |
| DRAM | ~60-90 ns | check transparent hugepages are actually applying |
| memory read | ~50 GB/s | - |
| jitter p99.99 | ~2 us | pinning missing, or emulator on a vCPU core |
| stalls >100us | 0-2 per 10 s | same causes; this is the tail an interactive session feels as hitching |
### Verifying hugepages actually apply
A guest can silently run entirely on 4 KiB pages. `memfd` with `shared` memory backing blocks
transparent hugepages, because shared mappings are not anonymous:
```sh
pid=$(pgrep -f "[q]emu-system-x86_64" | head -1)
sudo awk '/AnonHugePages/ {s+=$2} END {print s/1024 " MiB"}' /proc/$pid/smaps
```
Should be close to the guest's RAM size. It read **0 MiB** here until the leftover backing was
removed, at which point it read 8110 MiB of an 8 GiB guest.
That figure needs free, unfragmented host memory at the moment the guest touches its RAM. On the
same host with 6 GiB free, 14 GiB of page cache and `defrag` at `defer+madvise`, an 8 GiB guest
booted with 1154 MiB of its 7196 MiB resident on hugepages, and `/proc/vmstat` showed
`thp_fault_fallback` at 46% of `thp_fault_alloc`. Dropping the page cache and compacting before
the boot took it to 4064 MiB of 8261:
```sh
sudo sync; echo 3 | sudo tee /proc/sys/vm/drop_caches; echo 1 | sudo tee /proc/sys/vm/compact_memory
```
`vm-native-verify` prints the figure; without root it falls back to the host-wide
`AnonHugePages`, which is the guest's own number when only one guest runs. `vm-native-setup` says
when free memory is below the guest's RAM. Static hugepages would guarantee it and are deliberately
not used, because they take the memory from the host permanently.
Note the `[q]` in that pattern - an unbracketed `pgrep -f` matches your own shell and will hand you
the wrong PID.
---
## The selftest
`patches/kvm/0005-KVM-selftests-verify-ICEBP-DB-reports-RIP-past-the-ICEBP.patch` adds
`tools/testing/selftests/kvm/x86/icebp_test.c`. Build it in a kernel tree with the series applied:
```sh
make -C tools/testing/selftests/kvm "$PWD/tools/testing/selftests/kvm/x86/icebp_test"
sudo tools/testing/selftests/kvm/x86/icebp_test
```
On unpatched SVM it fails with `0x402ec4 != 0x402ec5 (db_rip != next_rip)`; with 0004 it passes.
It runs against whatever `kvm_amd` is loaded, so it is also the quickest way to tell which build
is live without booting the Windows guest.
## Reproducing the TIMER analysis
`bench/timerprobe.c` reproduces VMAware's instruction-latency detector so its ratio can be measured
directly, without needing a debug build of VMAware itself. It agrees with VMAware's own figure to
within a few percent, which is what makes it usable for iterating.
```sh
x86_64-w64-mingw32-gcc -O2 -o timerprobe.exe timerprobe.c
timerprobe.exe # sweep every counter placement, then detector 2
timerprobe.exe 0 2 # both detectors, measuring on cpu 0, counter on cpu 2
```
```
detector 1, instruction latency - counter placement swept:
counter=2 ratio= 7.896 (cpuid 2069 / lfence 262) DETECTED
detector 2, exception latency:
exception db= 8703.3 api= 5941.8 ratio= 1.465 pass (db traps seen: 4000)
```
With `cpuid_passthrough` on, detector 1 reads `cpuid 199 / lfence 262, ratio 0.760, pass`.
The sweep matters: it shows the best and worst case across placements, rather than only the one
VMAware happened to choose. That is how the cross-CCD idea was tested and ruled out.
Detector 2 is where the probe itself was wrong for two days. The first version used
`RaiseException` under a vectored handler for the software side, which never enters the kernel,
and reported a ratio of 6.169. VMAware's software side is `RtlCaptureContext` plus a direct
`ZwRaiseException` inside an SEH frame that executes its handler, so it pays a syscall, a kernel
dispatch and an `RtlUnwindEx`. The probe now does the same, with a `__C_specific_handler` scope
table because mingw has no `__try`, and its number agrees with VMAware's own debug line to within
a few percent. **Reproduce the mechanism, not the API name.**
---
## Host-side checks worth taking
```sh
# is the patched module actually the one loaded
cat /sys/module/kvm/srcversion
cat /sys/module/kvm_amd/srcversion
# did the CPU isolation hook fire, and did it restore
journalctl -t libvirt-cpu-isolation -n 10
# is the host whole again with the guest off
systemctl show --property=AllowedCPUs system.slice
nproc
free -g
```
The last group is not optional. A hook that confines the host and fails to restore is worse than
no hook, and it has happened here - a hung domain start left the host on twelve of thirty-two CPUs
with nothing running. Both the clean-shutdown and hard-destroy paths are now verified explicitly
rather than assumed.
---
## A rule learned the hard way
**Never call `virsh` from inside a libvirt hook.** libvirt blocks waiting for the hook to return
while the hook waits on libvirt. The deadlock wedges domain start, which wedges `virsh list`, which
is what virt-manager reads - so the symptom is virt-manager showing no VMs at all, which looks
nothing like the cause.