Performance: what the corrected guest costs

Benchmarks are bench/vmbench.c, cross-compiled with mingw-w64 and copied into the guest. Nothing is installed in the guest. No GPU is passed through, so every number here is a CPU, memory, scheduling or clock proxy for interactive and CPU-bound workloads - there is no graphics figure to be had.

Run scripts/verify-perf.sh to reproduce any of this.

The corrections are free

Ceiling is the same guest with the fidelity work abandoned entirely - hypervisor visible, the full Hyper-V enlightenment set, hypervclock. It is not a usable config, only a reference for how fast this guest can possibly go.

Conformant is the shipping config, the full level: 16 vCPU pinned to one CCD, hypervisor CPUID bit cleared, no enlightenments, patched QEMU, patched KVM modules, declaring a Ryzen 7 7700X. Scores 1/85, verdict "Running on bare metal".

Medians of 3 sequential runs each, patched modules in both, host governor performance:

	Conformant	Ceiling (detectable)	Difference
int latency, Mops	880	882	none
int throughput, Mops	5258	5294	-0.7%
fp, Mops	3512	3566	-1.5%
QPC cost, ns	15.1	14.5	none
jitter p99.99, us	1.7	1.4	none
stalls >100us / 10 s	0-2	0-1	none

Run-to-run spread inside each config (874-894 Mops conformant, 874-893 ceiling) is wider than the gap between them. The full correction set costs nothing measurable. There is no performance argument for leaving the guest uncorrected.

The Hyper-V enlightenments in particular buy nothing here. The one place they were expected to matter is the clock, and once the guest is sized to one CCD Windows uses the TSC for QPC by itself, leaving hypervclock nothing to improve.

The KVM patches are free too, and so is the CPUID passthrough

Measured 2026-09-05 on one boot each, same XML, three sequential vmbench all runs per row, the guest settled 60 s, nothing else running in it, host governor performance, the isolation hook confining the host to the twelve threads the guest does not use. Three module builds:

	`0001-0003`	`0001-0006`, passthrough off	`0001-0006`, passthrough on
int throughput, Mops	5308 / 5315 / 5318	5315 / 5315 / 5311	5256 / 5246 / 5250
QPC cost, ns	14.4 / 14.8 / 14.6	14.4 / 14.4 / 14.5	14.4 / 14.7 / 14.7
L3 latency, ns	10.26 / 10.19 / 10.20	10.21 / 10.29 / 10.35	10.40 / 10.39 / 10.39
jitter p99.99, us	1.4 / 1.3 / 1.3	1.4 / 1.4 / 1.4	1.4 / 1.4 / 1.4
stalls >100us / 10 s	0	1 / 0 / 0	0 / 0 / 0
DRAM read, GB/s	-	52.4 / 52.2 / 51.6	49.5 / 51.5 / 52.1

The passthrough-on column reads 1.2% lower on integer throughput, and every one of its three runs sits below every off run. That is exactly the shape a real cost would have, so it was re-run interleaved, off then on, three times, vmbench cpu only:

off 5285.5   on 5276.0
off 5285.5   on 5329.0
off 5329.3   on 5330.5

Gone. The first batch's gap was an order effect: it ran off-then-on straight after a QEMU build on the host, while the host load was still decaying. Benchmarks that share a machine with other work measure the sharing, even when the other work has just finished. Interleave, or the ordering becomes the result.

What the passthrough actually costs, measured from the host side:

vcpu_put rate: ~2300 context switches per second across the 16 vCPU threads, ~143 per vCPU, idle or under a one-thread load. Each one restores and re-applies the six brand-string MSRs.
Brand MSR writes: six WRMSR through /dev/cpu/8/msr take 791 ns against 712 ns for six RDMSR the same way, so at most ~130 ns per write including the syscall. Twelve writes per put/load cycle at 143 per second is under 0.03% of a vCPU.
Unpinned guest: the module withholds the passthrough from any vCPU thread whose allowed-CPU mask is wider than one CPU, checked on every entry, so the writes never happen. Measured with virsh vcpupin 0 0-31 --live on a running guest: the CPUID window goes from 199 ticks back to 2440, and returns to 199 when re-pinned.

A correction worth recording

An earlier version of this file reported a 3% cost for the corrections. That was wrong, and it was a measurement bug rather than a real effect: the repeat runs launched clock, cpu and jitter with &, so all three ran concurrently and competed for the same cores. Running them sequentially removes the gap. Benchmarks that share a machine with other benchmarks measure the sharing.

The finding that mattered

Windows' boot-time TSC calibration is a race, and losing it costs 85x on every timing call. QPC then costs ~1300 ns instead of ~15 for the rest of that boot. rdtsc stays cheap either way, so the TSC itself is fine; it is Windows' decision that changes.

What decides it is how much of the host the guest leaves alone. Four cold boots at each size, same XML:

Guest	Host keeps	Fast clock
24 vCPU	4 cores	4 / 4
28 vCPU	2 cores	3 / 4
32 vCPU	0 cores	2 / 4

An earlier version of this file reported a hard 16-vCPU ceiling. That was wrong twice over: it came from two data points with nothing tested between them, and one sample at each, when the behaviour is probabilistic rather than a threshold.

Ruled out by measurement, not assumption: the clock-source BCD settings (useplatformclock false, useplatformtick no, disabledynamictick yes) change nothing; invtsc changes nothing; pinning alone changes nothing; the cache-domain span changes nothing.

Realtime vCPU scheduling (<vcpusched scheduler='fifo'>) makes it actively worse: on a guest sized to the whole machine it failed to boot on two of three attempts and wedged the host.

The second-order effect is larger than the clock tax. Going from 32 to 16 vCPU takes stalls over 100 us from 452 per 10 s to 0-2, and p99.99 from 87 us to 1.6 us. That tail is what shows up as hitching in an interactive session, and mean throughput does not show it.

Dropping to 16 vCPU also gains ~5% single-thread and takes L3 latency from 17.4 to 10.2 ns, because 8 cores on one CCD share one L3 rather than straddling the Infinity Fabric.

Why the guest declares a Ryzen 7 7700X

16 vCPU contradicts a Ryzen 9 7950X, which is 32 threads, and VMAware's thread-count check reads the CPUID brand string - not SMBIOS, so the injected SMBIOS type 4 string is irrelevant to it.

The guest therefore declares a Ryzen 7 7700X, a real 8-core/16-thread Zen 4 part. The thread count is then correct and the check passes. It also makes the guest internally consistent: it genuinely is an 8-core Zen 4 chip on a single CCD, which is exactly what a 7700X is.

That is the thesis of the whole project in one example. The check does not stop firing because a better identity string was picked. It stops firing because the declared identity finally agrees with the silicon underneath it. Parts of a machine that contradict each other are exactly what a detection technique looks for.

Known limits of this measurement

c2c is not trustworthy and no conclusion is drawn from it. It reports ~34 ns for what should be an SMT pair, and moved 38 -> 127 ns between near-identical configs. The InterlockedExchange ping-pong's own overhead likely dominates.
cpu_mt at 8 threads is a harness artefact: it pins to CPUs 0-7, which in an 8c/2t guest are four cores' SMT pairs rather than eight cores, so it measures SMT contention, not scaling.
A background compute workload on the host is the most likely source of the occasional multi-millisecond outlier in the jitter tail, and it is not controlled for.
Hugepage coverage is a host-state figure, not a config figure: the same guest read 16% and 49% of resident memory on 2 MiB pages on two boots, depending on free memory and compaction before the start (docs/TESTING.md). None of the numbers above were taken with the coverage checked, which is one reason to trust the interleaved comparisons more than absolute values.
The diskio rows are not used for any conclusion: rand4k_IOPS moved between 5139 and 13650 across otherwise identical runs, which is the host page cache, not the guest.
The vm-native-verify sample quoted in the README and the docs page read 5012 Mops single-thread, 4% under the 5220-5330 the controlled runs above read on the same build. It was one run, taken a minute after boot with the guest still settling. vm-native-verify is a pass/fail gate, and its threshold is set for that; the cost claims here rest only on the interleaved runs.
The #GP intercept's individual cost has not been isolated; the three-column table above bounds the whole five-patch set at "within run-to-run spread".

8.6 KiB Raw Permalink Blame History