sandwich/vfio-native
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6777e6e3997031e9295c89f69a3acfc656b22de3
vfio-native/docs/ANALYSIS.md
T

29 KiB
Raw Blame History

Why each VMAware check fires, and what it took to clear it

The engineering record behind the fidelity work. README.md says what to apply and when; this says why, with the source references. Written 2026-09-03 at 2/85, revised 2026-09-05 at 1/85 on both VMAware v2.8.1 and VMAware HEAD (commit 95fecc2) - the one survivor is GPU_CAPABILITIES, and VMAware's own verdict is "Running on bare metal", likeliness 20%, confirmation false.

Source read against: VMAware src/vmaware.hpp, QEMU 11.1.2, kernel 7.2.3 arch/x86/kvm.

CLEARED - CPUID_SIGNATURE

Not about hypervisor leaves. The guest's 0x40000000 reads all zeros, and so does the bare-metal host, so it was never the trigger.

The AMD branch (cpuid_signature(), vmaware.hpp:6941-6970) reads leaf 7 subleaf 0, EDX and returns true if bit 26 (IBRS/IBPB), 27 (STIBP) or 31 (SSBD) is set. AMD reserves those three to zero and enumerates its mitigations in 0x80000008.EBX instead.

Measured with an unprivileged CPUID probe:

leaf 7.0 EDX bits 26/27/31 0x80000008.EBX 12/14/15/24/25
host, bare metal 0x10000010 0 0 0 1 1 1 1 1
guest, before 0x9c000010 set passed through
guest, after 0x30000010 0 0 0 1 1 1 1 1

KVM synthesises the Intel-style SPEC_CTRL interface on AMD, and host-passthrough enables it. Three <feature policy='disable'> lines fix it. All three are needed - the check ORs the bits.

Verified in-guest afterwards that every AMD-native mitigation bit survives, so the guest keeps the interface Windows actually uses on AMD. Residual: KVM also synthesises bit 29 (arch-capabilities), which bare metal lacks. Nothing reads it, so it is left alone.

CLEARED - SVM_EXCEPTIONS

A bug KVM documents against itself.

svm.c:272 is unconditional:

svm->vmcb->save.efer = efer | EFER_SVME;

Hardware therefore never sees the guest's real SVME=0. A CPL3 VMLOAD passes the SVME check, hardware notices CPL is not zero, and injects #GP before the VMLOAD intercept can fire. Bare metal raises #UD, and #UD is the one exception svm_exceptions() treats as innocent (vmaware.hpp:14551).

There is a FIXME at svm.c:1076-1080 describing exactly this.

The diagnostic that pinned it: the scored run printed [ DETECTED ] with no debug line. Both noisy exits in the function call vma_debug, so it must have fallen through to the bare return true at vmaware.hpp:14570 - which forces svm_visible == true and a non-#UD fault.

Fix, in svm_recalc_instruction_intercepts():

	if (!(vcpu->arch.efer & EFER_SVME)) {
		svm_set_intercept(svm, INTERCEPT_VMLOAD);
		svm_set_intercept(svm, INTERCEPT_VMSAVE);
		svm_set_intercept(svm, INTERCEPT_CLGI);
		svm_set_intercept(svm, INTERCEPT_STGI);
		set_exception_intercept(svm, GP_VECTOR);   /* added */
	} else {

gp_interception() then decodes the SVM opcode, sees !is_guest_mode, and calls svm_invoke_exit_handler(SVM_EXIT_VMLOAD) -> vmload_interception -> nested_svm_check_permissions (nested.c:1664), which queues UD_VECTOR.

The shipped 0001 also keeps the intercept across a guest clearing SVME later (the stock svm_set_efer() dropped it there), drops it again on SVME=1 where the erratum workaround does not need it, and never arms it for SEV guests, whose instructions KVM cannot decode - the same exclusion the erratum path already makes.

Do NOT disable svm in guest CPUID as an alternative. It does not change the exception, and it moves the check into its !svm_visible branch, which is worse - "SVM hypervisor hiding CPU capabilities", weight 150.

Do NOT use kvm.enable_vmware_backdoor=1 as the no-rebuild shortcut. It arms the same intercept, but permanently opens two well-known VMware detection surfaces in every guest on the host: emulate.c:2562 stops faulting CPL3 IN/OUT on ports 0x5658/0x5659, and emulate.c:3905 makes CPL3 RDPMC of pseudo-counters 0x10000-0x10002 return host TSC, which is architecturally impossible on real silicon. VMAware is blind to both. pafish, al-khaser and other detection tooling are not.

CLEARED - FIRMWARE

A chain that reports only its first hit. firmware() (vmaware.hpp:8860) runs six sections in fixed order against each firmware buffer, DSDT first, and the first match returns. Fixing one link costs a full QEMU rebuild and boot and only reveals the next - so bundle.

Links, in order:

  1. SMI Resources reservation string - _UID renamed
  2. PRTP/PRTA routing symmetry - renamed IRQP/IRQA
  3. Sequential PIRQ names, vmaware.hpp:9142 - GSIA-H -> APCA-H
  4. PNP0A06 resource stubs, vmaware.hpp:9148 - GPER -> RSRA, PHPR -> RSRB
  5. FACP C2/C3 latencies, vmaware.hpp:9242 - 0xfff -> 0xffe
  6. Debug Port OperationRegion at 0x0402 - already dead on this tree
  7. HPET register-validation loop - dormant, domain sets hpet present='no'. Re-arms if enabled.
  8. Dummy SATA, DMAR, APIC source overrides - dead on this tree (SATA at 1f.2: see the SATA subsection below - device 31's functions carry no generic _ADR node, so the D0FA+0x1F0002 pair never forms)

Link 3 requires all four of LNKE, LNKH, GSIE, GSIH, which is QEMU's eight-link plus eight-GSI layout. Real boards have four PCI link devices and no GSI-named ones, so rename the GSI* half and leave LNK* alone. Those names are declared by build_gsi_link_dev() and referenced only through build_q35_routing_table("GSI"), which derives every name from one 3-character prefix - so a prefix change moves every reference with it.

Do not pick IRQ as the replacement: _SB.PCI0.IRQA is already the APIC routing package and a bare NameSeg would bind to it.

0xffe keeps "C-state not supported" semantics (anything above 100 / 1000 means unsupported) while not matching the check's exact-equality test against 0x0FFF.

WAET must stay out. "WAET" is target index 18 in the section-2 scan and the table's signature sits at offset 0, i.e. an instant hit. An earlier sed-based patcher claimed it restored WAET and never did: build_waet() has zero call sites, so its grep -q build_waet guard was satisfied by the definition alone and the insert never ran. Guard on a call site, not on a definition.

The chain grew. firmware() returns on its first hit, so clearing one link only exposes the next, and current VMAware has added links past the eight above. 09-firmware-chain clears the rest, measured to 0 detections on both v2.8.1 and HEAD:

Link Fix
IQST/IQCR + PRR0/PRRI interrupt-query methods renamed PSTA/PCRS/PRS0/PRSI
TPM PPI regions/method TPP2/TPP3/TPFN renamed PPI2/PPI3/PPIF
zeroed _S5 SLP_TYP package 2-element package form
ACPI0006 _HID in \_GPE declared by full path \_GPE._HID; dropping it stops macOS powering off
Q35 MMCONFIG reservation device MCFG renamed PCMM
synthetic GSI link devices, empty _DIS/_SRS gave the no-op methods a body
PCIe _OSC And(CDW3, 0x1F, ..) mask swapped the And() operands (commutative)
PIRQ link _PRS fixed IRQ 5/10/11 listed the same set as {11, 10, 5}

A q35 guest also needs a named USB controller in the DSDT (the ICH9 EHCI/UHCI set, declared EHC1/ UHC1), or the release flags "complete absence of USB controller declarations"; that is a domain choice, not a QEMU patch.

CLEARED - the CPU-hotplug I/O signature that macOS forced back in (11-macos-cpu-hotplug)

macOS references \_SB.CPUS.Cxxx from its own CpuPlug SSDT, so the guest must carry QEMU's build_cpus_aml output (the \_SB.PCI0.PRES controller plus the \_SB.CPUS Processor objects) or it hangs in ACPI namespace init. 01-firmware had dropped that whole block, which is why macOS lived in an opt-in patch: the block reintroduces exactly one FIRMWARE tell, so a macOS-capable build scored 2/85 on v2.8.1 while HEAD stayed clean.

The tell is a single check. v2.8.1 firmware() carries cpu_hotplug_io[] = {0x0B, 0xD8, 0x0C} and byte-matches it anywhere in the tables. Those three bytes are WordData 0x0CD8 - the CPU-hotplug register port that build_cpus_aml writes into the PRST OperationRegion base (ICH9_CPU_HOTPLUG_IO_BASE). Confirmed by dumping the macOS-build DSDT: the bytes appear only at the OperationRegion operand, never in the _CRS. A small-resource IO descriptor encodes 0x0CD8 as 0x47 0x01 0xD8 0x0C 0xD8 0x0C 0x01 0x0C - the 0xD8 0x0C runs are preceded by 0x01/0x0C, never 0x0B - so the _CRS is not what matches and does not need touching. The renamed _UID ("Reserved resources C") already dodges the separate "CPU Hotplug resources" string check; it was never the live tell. HEAD has no cpu_hotplug_io check, so it never flagged the block.

Relocating the base does not work. The obvious fix - move ICH9_CPU_HOTPLUG_IO_BASE off 0x0CD8 so the pattern changes - clears the scanner but hangs every SMM/OVMF guest: macOS spins ~10 cores at boot and never reaches login, Windows the same. OvmfPkg/Include/IndustryStandard/Q35MchIch9.h hardcodes ICH9_CPU_HOTPLUG_BASE 0x0CD8 and OVMF's SMM code reads the register block at that fixed port, independent of the AML. The non-SMM path (plain OVMF, a bare Linux boot) survives a move; the SMM guests we actually run do not. A/B confirmed it: macOS reaches login on 0x0CD8, spins on 0x0CE4. Patching OVMF is already on the rejected list.

The fix keeps the port and hides only the AML operand. 11 emits the OperationRegion base for a SystemIO region as Add(0x0C00, 0xD8) instead of the literal 0x0CD8. It evaluates to the identical 0x0CD8 at namespace load - the port QEMU decodes, the port in _CRS, and the port OVMF reads are all unchanged - but the AML is 0x72 0x0B 0x00 0x0C 0x0A 0xD8 .., so the 0x0B 0xD8 0x0C run never appears. macOS's interpreter evaluates the computed base without complaint.

Verified on QEMU 11.1.2 with this fix in the default series: the DSDT keeps \_SB.CPUS/PRES/ Processor(C000..) and 0x0CD8, with no 0x0B 0xD8 0x0C anywhere (iasl-clean); macOS boots to login and powers off via S5 in ~12 s; VMAware v2.8.1 and HEAD both report detected=0 / FIRMWARE=0 on the stealth guest; Windows (secboot + SMM) and Linux boot unchanged. The opt-in split is gone.

CLEARED - the SATA controller OVMF must find at 1f.2 (03-pci-ids, 1.4.1)

03-pci-ids originally relocated the ICH9 SATA controller off its standard slot (ICH9_SATA1_DEV 31 -> 20) to move the dummy-SATA DSDT node away from 00:1f.2. That breaks OVMF first-boot for any SATA-booted guest: OVMF expects the q35 AHCI at 00:1f.2, so with it at 00:14.2 OVMF never builds the disk boot option and falls through to Start PXE over IPv4. Only macOS exposed this - it boots OpenCore off AHCI; the Windows identity guests boot from NVMe and never touched the moved controller. A/B confirmed it: stock QEMU reaches the OpenCore picker, the 1.4.0 patched binary PXEs, and restoring ICH9_SATA1_DEV = 31 reaches the picker again (BdsDxe: loading ... from PciRoot(0x0)/Pci(0x1F,0x2)/Sata).

The relocation was never load-bearing for the score. The dummy-SATA FIRMWARE link (vmaware.hpp ~9963) fires only on both the NameSeg D0FA and Name(_ADR, 0x001F0002). With SATA back at 1f.2, neither is present: QEMU describes device 31's functions under the ICH9 LPC/ISA scope, not through build_append_pci_bus_devices, so the controller gets no generic _ADR node at all, and get_mimic_pci_name never emits D0FA for any devfn. Dumped the DSDT of a 1f.2-SATA guest on the fixed build: zero D0FA, zero 08 5F 41 44 52 0C 02 00 1F 00. No vendor or device ID changes, so DEVICES / PCI_VENDOR / CPU_HEURISTIC are untouched.

Fix: drop the ICH9_SATA1_DEV hunk, leave SATA at stock 1f.2. macOS boots on the default series; 0/85 preserved.

CLEARED - the dangling ACPI name macOS aborts the DSDT on (folded into 01-firmware)

01-firmware renames the cold-plug PCI device ACPI nodes through get_mimic_pci_name (build_append_pci_bus_devices) to break the stock S+hex NameSeg signature. It left the hotplug and bus-notification code naming/referencing the same devices by the old S%.02X scheme, in four spots: the child-bridge Scope and ^S%.02X.PCNT cross-reference in build_append_notification_callback, the Notify() target in build_append_pcihp_notify_entry, and the per-slot Scope/Device in build_append_pcihp_slots. So the DSDT opens Scope(S13) on a device actually declared GFX3 - a dangling NameSeg.

Only macOS exposes it. Its ACPI parser aborts the whole DSDT load on the missing name: [S13_] Namespace lookup failure -> [DSDT] table load failed -> [\_SB.PCI0] ... AE_NOT_FOUND, then it hangs in early IOKit (last line AppleACPIPlatformExpert/cpus). Linux and Windows swallow the AE_NOT_FOUND and boot on, which is why the identity guests never showed it - only a from-scratch macOS recovery boot did. Diagnosed with macOS IOKit verbose (boot-args io=0xffffffff).

Fix (in 01-firmware, alongside the get_mimic_pci_name definition it completes): route those four sites through get_mimic_pci_name so definitions and references agree. It removes S%.02X NameSegs rather than adding any, so the score is unaffected - re-measured on the rebuilt binary, VMAware v2.8.1 and HEAD both detected=0/FIRMWARE=0, and a Tahoe recovery boots to the installer GUI.

CLEARED - CTYPE and SMBIOS_VM_BIT (the SMBIOS QEMU synthesises)

On Linux, VMAware reads SMBIOS four ways: DMIDECODE and DMI_SCAN match VM vendor strings in dmidecode -t system and seven /sys/class/dmi/id files, SMBIOS_VM_BIT tests bit 4 of type 0 byte 19, and CTYPE fires on chassis type 1, Other (HEAD e1a9786, vmaware.hpp:8272, :8948, :9023, :8234). On Windows, FIRMWARE also scans the raw tables (RSMB, :10369) for pc-q35 and Q35 + (:9726). The strings are domain identity. The bit, the chassis type and the default strings are what QEMU emits by itself.

QEMU synthesises every required structure the domain does not supply. Switching that off (smbios_defaults = false) leaves only what the domain passes: either a complete real dump through -smbios file=, or a table with no processor, memory or end-of-table structures. 10-smbios keeps it on and fixes what the synthesised structures leak:

Leak Fix
type 3 chassis type hard-coded to 1 (Other) defaults to 3 (Desktop), set with -smbios type=3,chassis-type=N
type 0 VM bit set whenever type 0 fields are given vm defaults off; libvirt <sysinfo> cannot pass vm=off
unset product and version strings Standard PC (Q35 + ICH9, 2009), pc-q35-<ver> Unknown, like the manufacturer from 04-fw-cfg

Measured on a Linux q35 guest, the same score on VMAware v2.8.1 and HEAD:

SMBIOS from the domain QEMU Score
complete real dump, 52 file= structures defaults off 0/85
the same dump defaults on 0/85, dmidecode byte-identical
field arguments for types 0-4 with vm=off defaults on, builders unpatched 1/85, CTYPE
field arguments for types 0-4, no vm=off, chassis-type set 10-smbios 0/85

Identity is authored as field arguments. A dump through -smbios file= adds what field arguments cannot express (memory module detail, OEM structures, battery) and mixes with them per type: a type given as file= cannot also take field arguments. Windows and macOS guests are not yet measured against this patch.

OPEN - GPU_CAPABILITIES (weight 20)

Not a GPU check. Four lines: GetDC(nullptr), then GetDeviceCaps(hdc, COLORMGMTCAPS), DETECTED if the result lacks CM_GAMMA_RAMP (0x2) or is CM_NONE. No adapter enumeration, no DXGI, no WMI, no device blacklist, no EDID - so the patched SAM/SyncMaster EDID override is irrelevant to this check.

A plausible theory said it was a harness artefact: scoring runs over SSH, OpenSSH on Windows lands in session 0, and a non-interactive window station has no gamma LUT. Tested and disproven. Re-ran as a scheduled task with /it, confirmed SessionId=1 and ScreenBounds={0,0,640,480} - same score, still fires.

So it is genuine: <video><model type='none'/> leaves Windows on a stub display with no gamma ramp. Real GPU passthrough fixes it. Out of scope without a spare card to pass through; vm-native-gpu is the path.

An emulated adapter is not a workaround: qxl and virtio-gpu are on the DEVICES blacklist (weight 100), and any display device can arm BOOT_LOGO (weight 90, brands QEMU) by giving OVMF a GOP.

CLEARED - KVM_INTERCEPTION

The debug string is a red herring, and it cost this project two wrong turns. It says "KVM attempting to patch instructions on the fly", which points straight at KVM_X86_QUIRK_FIX_HYPERCALL_INSN. That is not the cause.

VMAware runs its stubs at CPL3. KVM's emulator declares VMCALL in group7_rm0[1] (emulate.c:3971) as:

I(SrcNone | Priv | EmulateOnUD, em_hypercall),

Priv but no PrivUD. So emulate.c:5348:

	if ((ctxt->d & Priv) && ops->cpl(ctxt)) {
		if (ctxt->d & PrivUD)
			rc = emulate_ud(ctxt);
		else
			rc = emulate_gp(ctxt, 0);
		goto done;
	}

takes the #GP(0) branch, which Windows surfaces as STATUS_ACCESS_VIOLATION. The quirk code in emulator_fix_hypercall() sits downstream of that CPL check and is never reached. Disabling the quirk is a measured no-op - confirmed on three purpose-built /dev/kvm harnesses showing quirk-on and quirk-off producing byte-identical #GP with unmodified code bytes.

The earlier "cleared INTERCEPT_VMMCALL, effect: none" log entry was therefore correct, and my own mid-session reinterpretation of it as "closed for a bad reason" was wrong.

Both stubs need to raise #UD, and either half alone leaves the detection standing - fix stub 0 only and the loop reaches stub 1, whose silent -KVM_EPERM trips the generic-hypervisor branch instead. That is why every previous single-sided attempt measured as "no effect".

Half A. PrivUD (emulate.c:171, ((u64)1 << 51), "#UD instead of #GP on CPL > 0") added to the VMCALL entry in group7_rm0. The flag had zero users in the tree before this. Priv already routes to the CPL check; PrivUD picks the other branch. RIP does not advance.

Half B. kvm_emulate_hypercall() injects #UD for a CPL>0 hypercall instead of the silent -KVM_EPERM. Placed after the Xen and Hyper-V dispatch, so enlightened guests never reach it - and kvm_hv_hypercall already does the identical cpl != 0 -> kvm_queue_exception(UD_VECTOR) at hyperv.c:2546, so this makes the KVM-PV path consistent with the Hyper-V one rather than inventing new behaviour.

The blunt alternative - clearing INTERCEPT_VMMCALL - was correctly rejected: it breaks CPL0 hypercalls and bugchecks the Hyper-V domains.

Blast radius checked on the live host: svm_patch_hypercall writes VMMCALL and issues it at CPL0; Linux paravirt alternative-patches to VMMCALL, also CPL0; every KVM_HC_* already failed at CPL>0, so only the shape of the failure changes. Confirmed empirically by booting a Hyper-V-enlightened Windows guest and a Linux paravirt guest on the patched module - both boot and execute normally.

Half A ships as patches/kvm/0002-KVM-x86-emulator-UD-not-GP-for-VMCALL-at-CPL-0.patch, half B as patches/kvm/0003-KVM-x86-UD-for-hypercalls-issued-at-CPL-0.patch. Both land in kvm.ko, not kvm-amd.ko, so verify /sys/module/kvm/srcversion - checking only kvm_amd reports success on a stale build.

CLEARED - TIMER

There is no clock to lie to. The check does not use the TSC. It times a bare CPUID (EAX=0) against eight _mm_lfence(), using a second thread on another core spinning counter++ on a 64-byte-aligned volatile; the "tick" is a cross-core cache-line bounce (the code says so at vmaware.hpp:7480, "this is a cache-based counter").

serialize_available = cpu::is_intel() is hard-false on AMD, so the LFENCE branch always runs. Threads are pinned inside the guest with SetThreadGroupAffinity. The estimator takes a 500-1000 sample batch per trial, an interquartile mean of the middle 50%, then the minimum across 5 trials, with samples bracketed by counter-edge spin-waits and followed by burn_random_cycles() so a hypervisor cannot predict when to freeze the counter thread.

There are TWO detectors, and they OR together (vmaware.hpp:7484, 7488, 7493). Clearing one achieves nothing.

Detector 2 was never the wall - the probe was wrong

The first version of bench/timerprobe.c reported the exception-latency detector at 6.169 against a threshold of 2.5, and two days of work treated that as a hypervisor floor. It was a measurement error. VMAware's software side is RtlCaptureContext + ZwRaiseException inside an SEH frame with EXCEPTION_EXECUTE_HANDLER, which is a syscall in, a kernel dispatch, and an RtlUnwindEx out. The probe used RaiseException under a vectored handler, which never enters the kernel on the software side, so its reference window was several times too short and the ratio several times too high.

Rebuilt mechanism for mechanism (__C_specific_handler scope table, ZwRaiseException, EXCEPTION_EXECUTE_HANDLER), the probe reads 1.47, and VMAware's own debug line agrees on the same boot with the stock #DB intercept in place:

TIMER: Exception > VMM -> 10167 | nVMM -> 7066 | Ratio -> 1.439

Detector 2 passes on stock KVM. No bare-metal Windows baseline was taken - the brief asked for one, no physical Windows machine was available - so the only claim made here is the measured one: the guest is under the threshold with #DB intercepted, by a margin of 1.1.

Detector 1 is a world switch, and the only way to not pay it is to not exit

Measured: cpuid 2054-2383 / lfence 262-310 ticks, ratio 7.6-8.2, threshold 2.5. The exit itself is ~1800 ticks, on the order of 400 ns, an ordinary Zen 4 world switch. Cross-CCD counter placement, the exit fastpath, TSC offsetting and CPUID-leaf overrides were all measured and all failed (table below).

EXPERIMENTAL-0006-runtime-cpuid-passthrough clears INTERCEPT_CPUID on every guest entry once cpuid_passthrough=1, so CPUID runs on the silicon. Three things follow from raw CPUID, and the patch handles each:

  • The brand string reverts to the host SKU. The AMD Processor Name String MSRs (0xC0010030-35) back CPUID 0x80000002-4 directly and are writable and per-thread on Zen 4, so the declared SKU is written on the pinned core at entry and restored in svm_vcpu_put(). THREAD_MISMATCH stays clear; VMAware's own thread database resolves 7700X to 16 threads.
  • Raw CPUID advertises RDPRU (0x80000008 EBX[4]), which KVM masks out of its own CPUID and intercepts with kvm_handle_invalid_op(), i.e. #UD. VMAware's INTERRUPT_SHADOW and SINGLE_STEP both have an RDPRU variant gated on exactly that bit, and both report the #UD as an "exception anomaly, hypervisor seems to be present with CPUID interception disabled". Two full scans fired both checks before this was understood, and an exact-stub reproducer with a kvm_exit trace showed the TF #DB after a native CPUID landing at the right RIP every time - the single-step path was never at fault. The RDPRU intercept is cleared together with the CPUID one; the guest then reads the pinned core's real MPERF/APERF.
  • Raw CPUID must only be enabled after boot. Windows enumerates KVM's synthetic leaf-1 bits (x2apic, tsc-deadline) during boot and hangs if they vanish mid-enumeration. The module parameter is runtime-toggled for this reason.

The intercept is also withheld from any vCPU thread whose allowed-CPU mask is wider than one CPU (cpumask_weight(current->cpus_ptr) == 1), so the brand override cannot leak onto a host core: the patch verifies the 1:1 pinning itself rather than trusting the operator.

Measured with the passthrough on, VMAware HEAD debug, console session:

TIMER: Instruction > VMM -> 219 | nVMM -> 290 | Ratio -> 0.755
TIMER: Exception   > VMM -> 10147 | nVMM -> 7551 | Ratio -> 1.344
VM detections: 1/85   ===== CONCLUSION: Running on bare metal =====

The #DB intercept is left in place. Clearing it too was measured (detector 2 drops to 0.96) and gains nothing the score needs, while it removes the single-step re-injection KVM relies on for its own NMI-window logic. An earlier note here blamed that clear for the INTERRUPT_SHADOW and SINGLE_STEP detections; that was wrong, RDPRU was the cause both times.

Measured and dead before the passthrough:

Attempt Result
BetterTiming TSC compensation no detection change, 6x slower boot (6.1 s -> 38.6 s CPU)
Hypervisor-Phantom CPUID override ratio halved to ~810 ticks, still detected, and added SINGLE_STEP (10 -> 11)
Clear INTERCEPT_CPUID from boot guest never boots
All four intercepts cleared triple fault, EFER=0, instant
CPUID exit fastpath in svm_exit_handlers_fastpath the fastpath runs after the world switch (svm.c:4429, called at :4664), so it cannot remove the exit that is being measured
Cross-CCD topology to grow the baseline the measured ceiling falls ~1.7x short, and VMAware picks its own nearby core pair anyway
Both of the above combined still short

CLEARED - DBVM (weight 150, VMAware HEAD only)

Added to VMAware after v2.8.1; the release build does not fire it, HEAD does, with DBVM: ICEBP failed to advance guest RIP. Named after a hypervisor-backed debugger, but the sub-check that fires is generic: it clears DR0-DR7 through NtSetContextThread, executes F1 C3 (icebp; ret) under SEH, and compares the exception context's RIP against stub + 1.

ICEBP (INT1) raises a trap-like #DB, so the frame RIP on hardware is the next instruction. The first theory here - that SVM's #DB exception intercept catches it and db_interception() re-injects it with RIP still on the F1 byte - was wrong, and it was wrong in a way that only a trace could show: on Zen 4 the exception intercept never fires for ICEBP at all, and every warm reproducer read the correct +1. What the kvm_exit trace of a cold selftest shows instead:

kvm_exit: reason npf rip 0x402ec4 ... intr_info 0x80000301     <- ICEBP at 0x402ec4
kvm_inj_exception: #DB [reinjected]

The #DB was being delivered when it hit a nested page fault on an unmapped IDT/handler page. The exit carries the pending #DB in EXITINTINFO with the saved RIP on the ICEBP, and svm_complete_interrupts() re-queues it as a plain hardware exception, which the next VMRUN injects at that RIP. So the guest sees the ICEBP's own address whenever the delivery touches a page KVM has not mapped yet - deterministic in a fresh VM, sporadic in a running Windows guest (page compaction and reclaim keep unmapping NPT entries), which is why the scanner hit it and the warm probes did not.

0004-KVM-SVM-intercept-ICEBP-and-skip-it-before-injecting-its-DB enables SVM's dedicated INTERCEPT_ICEBP, which fires before the #DB exists, skips the instruction with svm_skip_emulated_instruction() and queues the #DB with the same DR6 payload db_interception() uses. Once RIP has been advanced, an injection that is itself interrupted is re-injected with the advanced RIP. This is the shape VMX already has (is_icebp() in vmx/vmcs.h, skipped in handle_exception_nmi()).

Proof, at CPL0 with no Windows involved - a KVM selftest (x86/icebp_test.c, shipped as 5/5 of the series) executes icebp under a #DB handler that records the frame RIP:

unpatched svm.c:   0x402ec4 != 0x402ec5 (db_rip != next_rip)    FAIL
with 0004:         PASS

and the two full scans with 0004 in place no longer fire DBVM.

Method notes worth keeping

Score from the console session. OpenSSH on Windows puts you in session 0, the services session. Every score this project took for its first two days came from there. Use a scheduled task with /it and confirm (Get-Process -Id $PID).SessionId is 1 - schtasks /run reports success regardless of where the task actually ran.

One change per boot. The 70-second silent power-off from an ACPI namespace failure has no error message and no log line. The only way to attribute it is to have changed one thing.

Read the debug line, not the count. [ DETECTED ] with no debug line is not "cleared" - some targets return silently (Xen at vmaware.hpp:9174, BXPC at :9186).

Status

VMAware v2.8.1 debug:  1/85   GPU_CAPABILITIES         Running on bare metal
VMAware HEAD debug:    1/85   GPU_CAPABILITIES         Running on bare metal, likeliness 20%

with 0001-0004 in the modules, EXPERIMENTAL-0006 built in and enabled after boot on a 16 vCPU guest pinned 1:1 to one CCD, declaring a Ryzen 7 7700X. The remaining check needs a real GPU passed through, which needs a spare card. bench/timerprobe.c reproduces both TIMER detectors and bench/-adjacent probes for the others live in docs/TESTING.md.

Reference in New Issue View Git Blame Copy Permalink