155 lines
8.6 KiB
Markdown
155 lines
8.6 KiB
Markdown
# Performance: what the corrected guest costs
|
|
|
|
Benchmarks are `bench/vmbench.c`, cross-compiled with mingw-w64 and copied into the guest.
|
|
Nothing is installed in the guest. No GPU is passed through, so every number here is a CPU,
|
|
memory, scheduling or clock proxy for interactive and CPU-bound workloads - there is no graphics
|
|
figure to be had.
|
|
|
|
Run `scripts/verify-perf.sh` to reproduce any of this.
|
|
|
|
## The corrections are free
|
|
|
|
**Ceiling** is the same guest with the fidelity work abandoned entirely - hypervisor visible, the
|
|
full Hyper-V enlightenment set, `hypervclock`. It is not a usable config, only a reference for how
|
|
fast this guest can possibly go.
|
|
|
|
**Conformant** is the shipping config, the `full` level: 16 vCPU pinned to one CCD, hypervisor
|
|
CPUID bit cleared, no enlightenments, patched QEMU, patched KVM modules, declaring a Ryzen 7
|
|
7700X. Scores 1/85, verdict "Running on bare metal".
|
|
|
|
Medians of 3 sequential runs each, patched modules in both, host governor `performance`:
|
|
|
|
| | Conformant | Ceiling (detectable) | Difference |
|
|
| --- | --- | --- | --- |
|
|
| int latency, Mops | 880 | 882 | none |
|
|
| int throughput, Mops | 5258 | 5294 | -0.7% |
|
|
| fp, Mops | 3512 | 3566 | -1.5% |
|
|
| QPC cost, ns | 15.1 | 14.5 | none |
|
|
| jitter p99.99, us | 1.7 | 1.4 | none |
|
|
| stalls >100us / 10 s | 0-2 | 0-1 | none |
|
|
|
|
Run-to-run spread inside each config (874-894 Mops conformant, 874-893 ceiling) is wider than the
|
|
gap between them. **The full correction set costs nothing measurable.** There is no performance
|
|
argument for leaving the guest uncorrected.
|
|
|
|
The Hyper-V enlightenments in particular buy nothing here. The one place they were expected to
|
|
matter is the clock, and once the guest is sized to one CCD Windows uses the TSC for QPC by
|
|
itself, leaving `hypervclock` nothing to improve.
|
|
|
|
### The KVM patches are free too, and so is the CPUID passthrough
|
|
|
|
Measured 2026-09-05 on one boot each, same XML, three sequential `vmbench all` runs per row, the
|
|
guest settled 60 s, nothing else running in it, host governor `performance`, the isolation hook
|
|
confining the host to the twelve threads the guest does not use. Three module builds:
|
|
|
|
| | `0001-0003` | `0001-0006`, passthrough off | `0001-0006`, passthrough on |
|
|
| --- | --- | --- | --- |
|
|
| int throughput, Mops | 5308 / 5315 / 5318 | 5315 / 5315 / 5311 | 5256 / 5246 / 5250 |
|
|
| QPC cost, ns | 14.4 / 14.8 / 14.6 | 14.4 / 14.4 / 14.5 | 14.4 / 14.7 / 14.7 |
|
|
| L3 latency, ns | 10.26 / 10.19 / 10.20 | 10.21 / 10.29 / 10.35 | 10.40 / 10.39 / 10.39 |
|
|
| jitter p99.99, us | 1.4 / 1.3 / 1.3 | 1.4 / 1.4 / 1.4 | 1.4 / 1.4 / 1.4 |
|
|
| stalls >100us / 10 s | 0 | 1 / 0 / 0 | 0 / 0 / 0 |
|
|
| DRAM read, GB/s | - | 52.4 / 52.2 / 51.6 | 49.5 / 51.5 / 52.1 |
|
|
|
|
The passthrough-on column reads 1.2% lower on integer throughput, and every one of its three runs
|
|
sits below every off run. That is exactly the shape a real cost would have, so it was re-run
|
|
interleaved, off then on, three times, `vmbench cpu` only:
|
|
|
|
off 5285.5 on 5276.0
|
|
off 5285.5 on 5329.0
|
|
off 5329.3 on 5330.5
|
|
|
|
Gone. The first batch's gap was an order effect: it ran off-then-on straight after a QEMU build on
|
|
the host, while the host load was still decaying. **Benchmarks that share a machine with other
|
|
work measure the sharing, even when the other work has just finished.** Interleave, or the
|
|
ordering becomes the result.
|
|
|
|
What the passthrough actually costs, measured from the host side:
|
|
|
|
- **vcpu_put rate**: ~2300 context switches per second across the 16 vCPU threads, ~143 per vCPU,
|
|
idle or under a one-thread load. Each one restores and re-applies the six brand-string MSRs.
|
|
- **Brand MSR writes**: six `WRMSR` through `/dev/cpu/8/msr` take 791 ns against 712 ns for six
|
|
`RDMSR` the same way, so at most ~130 ns per write including the syscall. Twelve writes per
|
|
put/load cycle at 143 per second is under 0.03% of a vCPU.
|
|
- **Unpinned guest**: the module withholds the passthrough from any vCPU thread whose allowed-CPU
|
|
mask is wider than one CPU, checked on every entry, so the writes never happen. Measured with
|
|
`virsh vcpupin 0 0-31 --live` on a running guest: the CPUID window goes from 199 ticks back to
|
|
2440, and returns to 199 when re-pinned.
|
|
|
|
### A correction worth recording
|
|
|
|
An earlier version of this file reported a 3% cost for the corrections. That was wrong, and it was
|
|
a measurement bug rather than a real effect: the repeat runs launched `clock`, `cpu` and `jitter`
|
|
with `&`, so all three ran concurrently and competed for the same cores. Running them
|
|
sequentially removes the gap. Benchmarks that share a machine with other benchmarks measure the
|
|
sharing.
|
|
|
|
## The finding that mattered
|
|
|
|
**Windows' boot-time TSC calibration is a race, and losing it costs 85x on every timing call.**
|
|
QPC then costs ~1300 ns instead of ~15 for the rest of that boot. `rdtsc` stays cheap either way,
|
|
so the TSC itself is fine; it is Windows' decision that changes.
|
|
|
|
What decides it is how much of the host the guest leaves alone. Four cold boots at each size,
|
|
same XML:
|
|
|
|
| Guest | Host keeps | Fast clock |
|
|
| --- | --- | --- |
|
|
| 24 vCPU | 4 cores | 4 / 4 |
|
|
| 28 vCPU | 2 cores | 3 / 4 |
|
|
| 32 vCPU | 0 cores | 2 / 4 |
|
|
|
|
An earlier version of this file reported a hard 16-vCPU ceiling. That was wrong twice over: it
|
|
came from two data points with nothing tested between them, and one sample at each, when the
|
|
behaviour is probabilistic rather than a threshold.
|
|
|
|
Ruled out by measurement, not assumption: the clock-source BCD settings
|
|
(`useplatformclock false`, `useplatformtick no`, `disabledynamictick yes`) change nothing;
|
|
`invtsc` changes nothing; pinning alone changes nothing; the cache-domain span changes nothing.
|
|
|
|
Realtime vCPU scheduling (`<vcpusched scheduler='fifo'>`) makes it actively worse: on a guest
|
|
sized to the whole machine it failed to boot on two of three attempts and wedged the host.
|
|
|
|
The second-order effect is larger than the clock tax. Going from 32 to 16 vCPU takes stalls over
|
|
100 us from 452 per 10 s to 0-2, and p99.99 from 87 us to 1.6 us. That tail is what shows up as
|
|
hitching in an interactive session, and mean throughput does not show it.
|
|
|
|
Dropping to 16 vCPU also gains ~5% single-thread and takes L3 latency from 17.4 to 10.2 ns,
|
|
because 8 cores on one CCD share one L3 rather than straddling the Infinity Fabric.
|
|
|
|
## Why the guest declares a Ryzen 7 7700X
|
|
|
|
16 vCPU contradicts a Ryzen 9 7950X, which is 32 threads, and VMAware's thread-count check reads
|
|
the **CPUID** brand string - not SMBIOS, so the injected SMBIOS type 4 string is irrelevant to it.
|
|
|
|
The guest therefore declares a Ryzen 7 7700X, a real 8-core/16-thread Zen 4 part. The thread count
|
|
is then correct and the check passes. It also makes the guest internally consistent: it genuinely
|
|
is an 8-core Zen 4 chip on a single CCD, which is exactly what a 7700X is.
|
|
|
|
That is the thesis of the whole project in one example. The check does not stop firing because a
|
|
better identity string was picked. It stops firing because the declared identity finally agrees
|
|
with the silicon underneath it. Parts of a machine that contradict each other are exactly what a
|
|
detection technique looks for.
|
|
|
|
## Known limits of this measurement
|
|
|
|
- `c2c` is not trustworthy and no conclusion is drawn from it. It reports ~34 ns for what should
|
|
be an SMT pair, and moved 38 -> 127 ns between near-identical configs. The
|
|
`InterlockedExchange` ping-pong's own overhead likely dominates.
|
|
- `cpu_mt` at 8 threads is a harness artefact: it pins to CPUs 0-7, which in an 8c/2t guest are
|
|
four cores' SMT pairs rather than eight cores, so it measures SMT contention, not scaling.
|
|
- A background compute workload on the host is the most likely source of the occasional
|
|
multi-millisecond outlier in the jitter tail, and it is not controlled for.
|
|
- Hugepage coverage is a host-state figure, not a config figure: the same guest read 16% and 49%
|
|
of resident memory on 2 MiB pages on two boots, depending on free memory and compaction before
|
|
the start (`docs/TESTING.md`). None of the numbers above were taken with the coverage checked,
|
|
which is one reason to trust the interleaved comparisons more than absolute values.
|
|
- The `diskio` rows are not used for any conclusion: `rand4k_IOPS` moved between 5139 and 13650
|
|
across otherwise identical runs, which is the host page cache, not the guest.
|
|
- The `vm-native-verify` sample quoted in the README and the docs page read 5012 Mops single-thread,
|
|
4% under the 5220-5330 the controlled runs above read on the same build. It was one run, taken
|
|
a minute after boot with the guest still settling. `vm-native-verify` is a pass/fail gate, and
|
|
its threshold is set for that; the cost claims here rest only on the interleaved runs.
|
|
- The `#GP` intercept's individual cost has not been isolated; the three-column table above
|
|
bounds the whole five-patch set at "within run-to-run spread".
|