sandwich/vfio-native
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c1e86768342d4b12d959ee92028899c2a13ffc82
vfio-native/docs/ANALYSIS.md

19 KiB
Raw Blame History

Why each VMAware check fires, and what it took to clear it

The engineering record behind the fidelity work. README.md says what to apply and when; this says why, with the source references. Written 2026-09-03 at 2/85, revised 2026-09-05 at 1/85 on both VMAware v2.8.1 and VMAware HEAD (commit 95fecc2) - the one survivor is GPU_CAPABILITIES, and VMAware's own verdict is "Running on bare metal", likeliness 20%, confirmation false.

Source read against: VMAware src/vmaware.hpp, QEMU 11.1.1, kernel 7.2.3 arch/x86/kvm.

CLEARED - CPUID_SIGNATURE

Not about hypervisor leaves. The guest's 0x40000000 reads all zeros, and so does the bare-metal host, so it was never the trigger.

The AMD branch (cpuid_signature(), vmaware.hpp:6941-6970) reads leaf 7 subleaf 0, EDX and returns true if bit 26 (IBRS/IBPB), 27 (STIBP) or 31 (SSBD) is set. AMD reserves those three to zero and enumerates its mitigations in 0x80000008.EBX instead.

Measured with an unprivileged CPUID probe:

leaf 7.0 EDX bits 26/27/31 0x80000008.EBX 12/14/15/24/25
host, bare metal 0x10000010 0 0 0 1 1 1 1 1
guest, before 0x9c000010 set passed through
guest, after 0x30000010 0 0 0 1 1 1 1 1

KVM synthesises the Intel-style SPEC_CTRL interface on AMD, and host-passthrough enables it. Three <feature policy='disable'> lines fix it. All three are needed - the check ORs the bits.

Verified in-guest afterwards that every AMD-native mitigation bit survives, so the guest keeps the interface Windows actually uses on AMD. Residual: KVM also synthesises bit 29 (arch-capabilities), which bare metal lacks. Nothing reads it, so it is left alone.

CLEARED - SVM_EXCEPTIONS

A bug KVM documents against itself.

svm.c:272 is unconditional:

svm->vmcb->save.efer = efer | EFER_SVME;

Hardware therefore never sees the guest's real SVME=0. A CPL3 VMLOAD passes the SVME check, hardware notices CPL is not zero, and injects #GP before the VMLOAD intercept can fire. Bare metal raises #UD, and #UD is the one exception svm_exceptions() treats as innocent (vmaware.hpp:14551).

There is a FIXME at svm.c:1076-1080 describing exactly this.

The diagnostic that pinned it: the scored run printed [ DETECTED ] with no debug line. Both noisy exits in the function call vma_debug, so it must have fallen through to the bare return true at vmaware.hpp:14570 - which forces svm_visible == true and a non-#UD fault.

Fix, in svm_recalc_instruction_intercepts():

	if (!(vcpu->arch.efer & EFER_SVME)) {
		svm_set_intercept(svm, INTERCEPT_VMLOAD);
		svm_set_intercept(svm, INTERCEPT_VMSAVE);
		svm_set_intercept(svm, INTERCEPT_CLGI);
		svm_set_intercept(svm, INTERCEPT_STGI);
		set_exception_intercept(svm, GP_VECTOR);   /* added */
	} else {

gp_interception() then decodes the SVM opcode, sees !is_guest_mode, and calls svm_invoke_exit_handler(SVM_EXIT_VMLOAD) -> vmload_interception -> nested_svm_check_permissions (nested.c:1664), which queues UD_VECTOR.

The shipped 0001 also keeps the intercept across a guest clearing SVME later (the stock svm_set_efer() dropped it there), drops it again on SVME=1 where the erratum workaround does not need it, and never arms it for SEV guests, whose instructions KVM cannot decode - the same exclusion the erratum path already makes.

Do NOT disable svm in guest CPUID as an alternative. It does not change the exception, and it moves the check into its !svm_visible branch, which is worse - "SVM hypervisor hiding CPU capabilities", weight 150.

Do NOT use kvm.enable_vmware_backdoor=1 as the no-rebuild shortcut. It arms the same intercept, but permanently opens two well-known VMware detection surfaces in every guest on the host: emulate.c:2562 stops faulting CPL3 IN/OUT on ports 0x5658/0x5659, and emulate.c:3905 makes CPL3 RDPMC of pseudo-counters 0x10000-0x10002 return host TSC, which is architecturally impossible on real silicon. VMAware is blind to both. pafish, al-khaser and other detection tooling are not.

CLEARED - FIRMWARE

A chain that reports only its first hit. firmware() (vmaware.hpp:8860) runs six sections in fixed order against each firmware buffer, DSDT first, and the first match returns. Fixing one link costs a full QEMU rebuild and boot and only reveals the next - so bundle.

Links, in order:

  1. SMI Resources reservation string - _UID renamed
  2. PRTP/PRTA routing symmetry - renamed IRQP/IRQA
  3. Sequential PIRQ names, vmaware.hpp:9142 - GSIA-H -> APCA-H
  4. PNP0A06 resource stubs, vmaware.hpp:9148 - GPER -> RSRA, PHPR -> RSRB
  5. FACP C2/C3 latencies, vmaware.hpp:9242 - 0xfff -> 0xffe
  6. Debug Port OperationRegion at 0x0402 - already dead on this tree
  7. HPET register-validation loop - dormant, domain sets hpet present='no'. Re-arms if enabled.
  8. Dummy SATA, DMAR, APIC source overrides - already dead on this tree

Link 3 requires all four of LNKE, LNKH, GSIE, GSIH, which is QEMU's eight-link plus eight-GSI layout. Real boards have four PCI link devices and no GSI-named ones, so rename the GSI* half and leave LNK* alone. Those names are declared by build_gsi_link_dev() and referenced only through build_q35_routing_table("GSI"), which derives every name from one 3-character prefix - so a prefix change moves every reference with it.

Do not pick IRQ as the replacement: _SB.PCI0.IRQA is already the APIC routing package and a bare NameSeg would bind to it.

0xffe keeps "C-state not supported" semantics (anything above 100 / 1000 means unsupported) while not matching the check's exact-equality test against 0x0FFF.

WAET must stay out. "WAET" is target index 18 in the section-2 scan and the table's signature sits at offset 0, i.e. an instant hit. An earlier sed-based patcher claimed it restored WAET and never did: build_waet() has zero call sites, so its grep -q build_waet guard was satisfied by the definition alone and the insert never ran. Guard on a call site, not on a definition.

OPEN - GPU_CAPABILITIES (weight 20)

Not a GPU check. Four lines: GetDC(nullptr), then GetDeviceCaps(hdc, COLORMGMTCAPS), DETECTED if the result lacks CM_GAMMA_RAMP (0x2) or is CM_NONE. No adapter enumeration, no DXGI, no WMI, no device blacklist, no EDID - so the patched SAM/SyncMaster EDID override is irrelevant to this check.

A plausible theory said it was a harness artefact: scoring runs over SSH, OpenSSH on Windows lands in session 0, and a non-interactive window station has no gamma LUT. Tested and disproven. Re-ran as a scheduled task with /it, confirmed SessionId=1 and ScreenBounds={0,0,640,480} - same score, still fires.

So it is genuine: <video><model type='none'/> leaves Windows on a stub display with no gamma ramp. Real GPU passthrough fixes it. Out of scope without a spare card to pass through; vm-native-gpu is the path.

An emulated adapter is not a workaround: qxl and virtio-gpu are on the DEVICES blacklist (weight 100), and any display device can arm BOOT_LOGO (weight 90, brands QEMU) by giving OVMF a GOP.

CLEARED - KVM_INTERCEPTION

The debug string is a red herring, and it cost this project two wrong turns. It says "KVM attempting to patch instructions on the fly", which points straight at KVM_X86_QUIRK_FIX_HYPERCALL_INSN. That is not the cause.

VMAware runs its stubs at CPL3. KVM's emulator declares VMCALL in group7_rm0[1] (emulate.c:3971) as:

I(SrcNone | Priv | EmulateOnUD, em_hypercall),

Priv but no PrivUD. So emulate.c:5348:

	if ((ctxt->d & Priv) && ops->cpl(ctxt)) {
		if (ctxt->d & PrivUD)
			rc = emulate_ud(ctxt);
		else
			rc = emulate_gp(ctxt, 0);
		goto done;
	}

takes the #GP(0) branch, which Windows surfaces as STATUS_ACCESS_VIOLATION. The quirk code in emulator_fix_hypercall() sits downstream of that CPL check and is never reached. Disabling the quirk is a measured no-op - confirmed on three purpose-built /dev/kvm harnesses showing quirk-on and quirk-off producing byte-identical #GP with unmodified code bytes.

The earlier "cleared INTERCEPT_VMMCALL, effect: none" log entry was therefore correct, and my own mid-session reinterpretation of it as "closed for a bad reason" was wrong.

Both stubs need to raise #UD, and either half alone leaves the detection standing - fix stub 0 only and the loop reaches stub 1, whose silent -KVM_EPERM trips the generic-hypervisor branch instead. That is why every previous single-sided attempt measured as "no effect".

Half A. PrivUD (emulate.c:171, ((u64)1 << 51), "#UD instead of #GP on CPL > 0") added to the VMCALL entry in group7_rm0. The flag had zero users in the tree before this. Priv already routes to the CPL check; PrivUD picks the other branch. RIP does not advance.

Half B. kvm_emulate_hypercall() injects #UD for a CPL>0 hypercall instead of the silent -KVM_EPERM. Placed after the Xen and Hyper-V dispatch, so enlightened guests never reach it - and kvm_hv_hypercall already does the identical cpl != 0 -> kvm_queue_exception(UD_VECTOR) at hyperv.c:2546, so this makes the KVM-PV path consistent with the Hyper-V one rather than inventing new behaviour.

The blunt alternative - clearing INTERCEPT_VMMCALL - was correctly rejected: it breaks CPL0 hypercalls and bugchecks the Hyper-V domains.

Blast radius checked on the live host: svm_patch_hypercall writes VMMCALL and issues it at CPL0; Linux paravirt alternative-patches to VMMCALL, also CPL0; every KVM_HC_* already failed at CPL>0, so only the shape of the failure changes. Confirmed empirically by booting a Hyper-V-enlightened Windows guest and a Linux paravirt guest on the patched module - both boot and execute normally.

Half A ships as patches/kvm/0002-KVM-x86-emulator-UD-not-GP-for-VMCALL-at-CPL-0.patch, half B as patches/kvm/0003-KVM-x86-UD-for-hypercalls-issued-at-CPL-0.patch. Both land in kvm.ko, not kvm-amd.ko, so verify /sys/module/kvm/srcversion - checking only kvm_amd reports success on a stale build.

CLEARED - TIMER

There is no clock to lie to. The check does not use the TSC. It times a bare CPUID (EAX=0) against eight _mm_lfence(), using a second thread on another core spinning counter++ on a 64-byte-aligned volatile; the "tick" is a cross-core cache-line bounce (the code says so at vmaware.hpp:7480, "this is a cache-based counter").

serialize_available = cpu::is_intel() is hard-false on AMD, so the LFENCE branch always runs. Threads are pinned inside the guest with SetThreadGroupAffinity. The estimator takes a 500-1000 sample batch per trial, an interquartile mean of the middle 50%, then the minimum across 5 trials, with samples bracketed by counter-edge spin-waits and followed by burn_random_cycles() so a hypervisor cannot predict when to freeze the counter thread.

There are TWO detectors, and they OR together (vmaware.hpp:7484, 7488, 7493). Clearing one achieves nothing.

Detector 2 was never the wall - the probe was wrong

The first version of bench/timerprobe.c reported the exception-latency detector at 6.169 against a threshold of 2.5, and two days of work treated that as a hypervisor floor. It was a measurement error. VMAware's software side is RtlCaptureContext + ZwRaiseException inside an SEH frame with EXCEPTION_EXECUTE_HANDLER, which is a syscall in, a kernel dispatch, and an RtlUnwindEx out. The probe used RaiseException under a vectored handler, which never enters the kernel on the software side, so its reference window was several times too short and the ratio several times too high.

Rebuilt mechanism for mechanism (__C_specific_handler scope table, ZwRaiseException, EXCEPTION_EXECUTE_HANDLER), the probe reads 1.47, and VMAware's own debug line agrees on the same boot with the stock #DB intercept in place:

TIMER: Exception > VMM -> 10167 | nVMM -> 7066 | Ratio -> 1.439

Detector 2 passes on stock KVM. No bare-metal Windows baseline was taken - the brief asked for one, no physical Windows machine was available - so the only claim made here is the measured one: the guest is under the threshold with #DB intercepted, by a margin of 1.1.

Detector 1 is a world switch, and the only way to not pay it is to not exit

Measured: cpuid 2054-2383 / lfence 262-310 ticks, ratio 7.6-8.2, threshold 2.5. The exit itself is ~1800 ticks, on the order of 400 ns, an ordinary Zen 4 world switch. Cross-CCD counter placement, the exit fastpath, TSC offsetting and CPUID-leaf overrides were all measured and all failed (table below).

EXPERIMENTAL-0006-runtime-cpuid-passthrough clears INTERCEPT_CPUID on every guest entry once cpuid_passthrough=1, so CPUID runs on the silicon. Three things follow from raw CPUID, and the patch handles each:

  • The brand string reverts to the host SKU. The AMD Processor Name String MSRs (0xC0010030-35) back CPUID 0x80000002-4 directly and are writable and per-thread on Zen 4, so the declared SKU is written on the pinned core at entry and restored in svm_vcpu_put(). THREAD_MISMATCH stays clear; VMAware's own thread database resolves 7700X to 16 threads.
  • Raw CPUID advertises RDPRU (0x80000008 EBX[4]), which KVM masks out of its own CPUID and intercepts with kvm_handle_invalid_op(), i.e. #UD. VMAware's INTERRUPT_SHADOW and SINGLE_STEP both have an RDPRU variant gated on exactly that bit, and both report the #UD as an "exception anomaly, hypervisor seems to be present with CPUID interception disabled". Two full scans fired both checks before this was understood, and an exact-stub reproducer with a kvm_exit trace showed the TF #DB after a native CPUID landing at the right RIP every time - the single-step path was never at fault. The RDPRU intercept is cleared together with the CPUID one; the guest then reads the pinned core's real MPERF/APERF.
  • Raw CPUID must only be enabled after boot. Windows enumerates KVM's synthetic leaf-1 bits (x2apic, tsc-deadline) during boot and hangs if they vanish mid-enumeration. The module parameter is runtime-toggled for this reason.

The intercept is also withheld from any vCPU thread whose allowed-CPU mask is wider than one CPU (cpumask_weight(current->cpus_ptr) == 1), so the brand override cannot leak onto a host core: the patch verifies the 1:1 pinning itself rather than trusting the operator.

Measured with the passthrough on, VMAware HEAD debug, console session:

TIMER: Instruction > VMM -> 219 | nVMM -> 290 | Ratio -> 0.755
TIMER: Exception   > VMM -> 10147 | nVMM -> 7551 | Ratio -> 1.344
VM detections: 1/85   ===== CONCLUSION: Running on bare metal =====

The #DB intercept is left in place. Clearing it too was measured (detector 2 drops to 0.96) and gains nothing the score needs, while it removes the single-step re-injection KVM relies on for its own NMI-window logic. An earlier note here blamed that clear for the INTERRUPT_SHADOW and SINGLE_STEP detections; that was wrong, RDPRU was the cause both times.

Measured and dead before the passthrough:

Attempt Result
BetterTiming TSC compensation no detection change, 6x slower boot (6.1 s -> 38.6 s CPU)
Hypervisor-Phantom CPUID override ratio halved to ~810 ticks, still detected, and added SINGLE_STEP (10 -> 11)
Clear INTERCEPT_CPUID from boot guest never boots
All four intercepts cleared triple fault, EFER=0, instant
CPUID exit fastpath in svm_exit_handlers_fastpath the fastpath runs after the world switch (svm.c:4429, called at :4664), so it cannot remove the exit that is being measured
Cross-CCD topology to grow the baseline the measured ceiling falls ~1.7x short, and VMAware picks its own nearby core pair anyway
Both of the above combined still short

CLEARED - DBVM (weight 150, VMAware HEAD only)

Added to VMAware after v2.8.1; the release build does not fire it, HEAD does, with DBVM: ICEBP failed to advance guest RIP. Named after a hypervisor-backed debugger, but the sub-check that fires is generic: it clears DR0-DR7 through NtSetContextThread, executes F1 C3 (icebp; ret) under SEH, and compares the exception context's RIP against stub + 1.

ICEBP (INT1) raises a trap-like #DB, so the frame RIP on hardware is the next instruction. The first theory here - that SVM's #DB exception intercept catches it and db_interception() re-injects it with RIP still on the F1 byte - was wrong, and it was wrong in a way that only a trace could show: on Zen 4 the exception intercept never fires for ICEBP at all, and every warm reproducer read the correct +1. What the kvm_exit trace of a cold selftest shows instead:

kvm_exit: reason npf rip 0x402ec4 ... intr_info 0x80000301     <- ICEBP at 0x402ec4
kvm_inj_exception: #DB [reinjected]

The #DB was being delivered when it hit a nested page fault on an unmapped IDT/handler page. The exit carries the pending #DB in EXITINTINFO with the saved RIP on the ICEBP, and svm_complete_interrupts() re-queues it as a plain hardware exception, which the next VMRUN injects at that RIP. So the guest sees the ICEBP's own address whenever the delivery touches a page KVM has not mapped yet - deterministic in a fresh VM, sporadic in a running Windows guest (page compaction and reclaim keep unmapping NPT entries), which is why the scanner hit it and the warm probes did not.

0004-KVM-SVM-intercept-ICEBP-and-skip-it-before-injecting-its-DB enables SVM's dedicated INTERCEPT_ICEBP, which fires before the #DB exists, skips the instruction with svm_skip_emulated_instruction() and queues the #DB with the same DR6 payload db_interception() uses. Once RIP has been advanced, an injection that is itself interrupted is re-injected with the advanced RIP. This is the shape VMX already has (is_icebp() in vmx/vmcs.h, skipped in handle_exception_nmi()).

Proof, at CPL0 with no Windows involved - a KVM selftest (x86/icebp_test.c, shipped as 5/5 of the series) executes icebp under a #DB handler that records the frame RIP:

unpatched svm.c:   0x402ec4 != 0x402ec5 (db_rip != next_rip)    FAIL
with 0004:         PASS

and the two full scans with 0004 in place no longer fire DBVM.

Method notes worth keeping

Score from the console session. OpenSSH on Windows puts you in session 0, the services session. Every score this project took for its first two days came from there. Use a scheduled task with /it and confirm (Get-Process -Id $PID).SessionId is 1 - schtasks /run reports success regardless of where the task actually ran.

One change per boot. The 70-second silent power-off from an ACPI namespace failure has no error message and no log line. The only way to attribute it is to have changed one thing.

Read the debug line, not the count. [ DETECTED ] with no debug line is not "cleared" - some targets return silently (Xen at vmaware.hpp:9174, BXPC at :9186).

Status

VMAware v2.8.1 debug:  1/85   GPU_CAPABILITIES         Running on bare metal
VMAware HEAD debug:    1/85   GPU_CAPABILITIES         Running on bare metal, likeliness 20%

with 0001-0004 in the modules, EXPERIMENTAL-0006 built in and enabled after boot on a 16 vCPU guest pinned 1:1 to one CCD, declaring a Ryzen 7 7700X. The remaining check needs a real GPU passed through, which needs a spare card. bench/timerprobe.c reproduces both TIMER detectors and bench/-adjacent probes for the others live in docs/TESTING.md.

Reference in New Issue View Git Blame Copy Permalink