docs: readme, analysis, results, testing and guest setup

docs/RESULTS.md

@@ -0,0 +1,154 @@
+# Performance: what the corrected guest costs
+
+Benchmarks are `bench/vmbench.c`, cross-compiled with mingw-w64 and copied into the guest.
+Nothing is installed in the guest. No GPU is passed through, so every number here is a CPU,
+memory, scheduling or clock proxy for interactive and CPU-bound workloads - there is no graphics
+figure to be had.
+
+Run `scripts/verify-perf.sh` to reproduce any of this.
+
+## The corrections are free
+
+**Ceiling** is the same guest with the fidelity work abandoned entirely - hypervisor visible, the
+full Hyper-V enlightenment set, `hypervclock`. It is not a usable config, only a reference for how
+fast this guest can possibly go.
+
+**Conformant** is the shipping config, the `full` level: 16 vCPU pinned to one CCD, hypervisor
+CPUID bit cleared, no enlightenments, patched QEMU, patched KVM modules, declaring a Ryzen 7
+7700X. Scores 1/85, verdict "Running on bare metal".
+
+Medians of 3 sequential runs each, patched modules in both, host governor `performance`:
+
+| | Conformant | Ceiling (detectable) | Difference |
+| --- | --- | --- | --- |
+| int latency, Mops | 880 | 882 | none |
+| int throughput, Mops | 5258 | 5294 | -0.7% |
+| fp, Mops | 3512 | 3566 | -1.5% |
+| QPC cost, ns | 15.1 | 14.5 | none |
+| jitter p99.99, us | 1.7 | 1.4 | none |
+| stalls >100us / 10 s | 0-2 | 0-1 | none |
+
+Run-to-run spread inside each config (874-894 Mops conformant, 874-893 ceiling) is wider than the
+gap between them. **The full correction set costs nothing measurable.** There is no performance
+argument for leaving the guest uncorrected.
+
+The Hyper-V enlightenments in particular buy nothing here. The one place they were expected to
+matter is the clock, and once the guest is sized to one CCD Windows uses the TSC for QPC by
+itself, leaving `hypervclock` nothing to improve.
+
+### The KVM patches are free too, and so is the CPUID passthrough
+
+Measured 2026-09-05 on one boot each, same XML, three sequential `vmbench all` runs per row, the
+guest settled 60 s, nothing else running in it, host governor `performance`, the isolation hook
+confining the host to the twelve threads the guest does not use. Three module builds:
+
+| | `0001-0003` | `0001-0006`, passthrough off | `0001-0006`, passthrough on |
+| --- | --- | --- | --- |
+| int throughput, Mops | 5308 / 5315 / 5318 | 5315 / 5315 / 5311 | 5256 / 5246 / 5250 |
+| QPC cost, ns | 14.4 / 14.8 / 14.6 | 14.4 / 14.4 / 14.5 | 14.4 / 14.7 / 14.7 |
+| L3 latency, ns | 10.26 / 10.19 / 10.20 | 10.21 / 10.29 / 10.35 | 10.40 / 10.39 / 10.39 |
+| jitter p99.99, us | 1.4 / 1.3 / 1.3 | 1.4 / 1.4 / 1.4 | 1.4 / 1.4 / 1.4 |
+| stalls >100us / 10 s | 0 | 1 / 0 / 0 | 0 / 0 / 0 |
+| DRAM read, GB/s | - | 52.4 / 52.2 / 51.6 | 49.5 / 51.5 / 52.1 |
+
+The passthrough-on column reads 1.2% lower on integer throughput, and every one of its three runs
+sits below every off run. That is exactly the shape a real cost would have, so it was re-run
+interleaved, off then on, three times, `vmbench cpu` only:
+
+    off 5285.5   on 5276.0
+    off 5285.5   on 5329.0
+    off 5329.3   on 5330.5
+
+Gone. The first batch's gap was an order effect: it ran off-then-on straight after a QEMU build on
+the host, while the host load was still decaying. **Benchmarks that share a machine with other
+work measure the sharing, even when the other work has just finished.** Interleave, or the
+ordering becomes the result.
+
+What the passthrough actually costs, measured from the host side:
+
+- **vcpu_put rate**: ~2300 context switches per second across the 16 vCPU threads, ~143 per vCPU,
+  idle or under a one-thread load. Each one restores and re-applies the six brand-string MSRs.
+- **Brand MSR writes**: six `WRMSR` through `/dev/cpu/8/msr` take 791 ns against 712 ns for six
+  `RDMSR` the same way, so at most ~130 ns per write including the syscall. Twelve writes per
+  put/load cycle at 143 per second is under 0.03% of a vCPU.
+- **Unpinned guest**: the module withholds the passthrough from any vCPU thread whose allowed-CPU
+  mask is wider than one CPU, checked on every entry, so the writes never happen. Measured with
+  `virsh vcpupin 0 0-31 --live` on a running guest: the CPUID window goes from 199 ticks back to
+  2440, and returns to 199 when re-pinned.
+
+### A correction worth recording
+
+An earlier version of this file reported a 3% cost for the corrections. That was wrong, and it was
+a measurement bug rather than a real effect: the repeat runs launched `clock`, `cpu` and `jitter`
+with `&`, so all three ran concurrently and competed for the same cores. Running them
+sequentially removes the gap. Benchmarks that share a machine with other benchmarks measure the
+sharing.
+
+## The finding that mattered
+
+**Windows' boot-time TSC calibration is a race, and losing it costs 85x on every timing call.**
+QPC then costs ~1300 ns instead of ~15 for the rest of that boot. `rdtsc` stays cheap either way,
+so the TSC itself is fine; it is Windows' decision that changes.
+
+What decides it is how much of the host the guest leaves alone. Four cold boots at each size,
+same XML:
+
+| Guest | Host keeps | Fast clock |
+| --- | --- | --- |
+| 24 vCPU | 4 cores | 4 / 4 |
+| 28 vCPU | 2 cores | 3 / 4 |
+| 32 vCPU | 0 cores | 2 / 4 |
+
+An earlier version of this file reported a hard 16-vCPU ceiling. That was wrong twice over: it
+came from two data points with nothing tested between them, and one sample at each, when the
+behaviour is probabilistic rather than a threshold.
+
+Ruled out by measurement, not assumption: the clock-source BCD settings
+(`useplatformclock false`, `useplatformtick no`, `disabledynamictick yes`) change nothing;
+`invtsc` changes nothing; pinning alone changes nothing; the cache-domain span changes nothing.
+
+Realtime vCPU scheduling (`<vcpusched scheduler='fifo'>`) makes it actively worse: on a guest
+sized to the whole machine it failed to boot on two of three attempts and wedged the host.
+
+The second-order effect is larger than the clock tax. Going from 32 to 16 vCPU takes stalls over
+100 us from 452 per 10 s to 0-2, and p99.99 from 87 us to 1.6 us. That tail is what shows up as
+hitching in an interactive session, and mean throughput does not show it.
+
+Dropping to 16 vCPU also gains ~5% single-thread and takes L3 latency from 17.4 to 10.2 ns,
+because 8 cores on one CCD share one L3 rather than straddling the Infinity Fabric.
+
+## Why the guest declares a Ryzen 7 7700X
+
+16 vCPU contradicts a Ryzen 9 7950X, which is 32 threads, and VMAware's thread-count check reads
+the **CPUID** brand string - not SMBIOS, so the injected SMBIOS type 4 string is irrelevant to it.
+
+The guest therefore declares a Ryzen 7 7700X, a real 8-core/16-thread Zen 4 part. The thread count
+is then correct and the check passes. It also makes the guest internally consistent: it genuinely
+is an 8-core Zen 4 chip on a single CCD, which is exactly what a 7700X is.
+
+That is the thesis of the whole project in one example. The check does not stop firing because a
+better identity string was picked. It stops firing because the declared identity finally agrees
+with the silicon underneath it. Parts of a machine that contradict each other are exactly what a
+detection technique looks for.
+
+## Known limits of this measurement
+
+- `c2c` is not trustworthy and no conclusion is drawn from it. It reports ~34 ns for what should
+  be an SMT pair, and moved 38 -> 127 ns between near-identical configs. The
+  `InterlockedExchange` ping-pong's own overhead likely dominates.
+- `cpu_mt` at 8 threads is a harness artefact: it pins to CPUs 0-7, which in an 8c/2t guest are
+  four cores' SMT pairs rather than eight cores, so it measures SMT contention, not scaling.
+- A background compute workload on the host is the most likely source of the occasional
+  multi-millisecond outlier in the jitter tail, and it is not controlled for.
+- Hugepage coverage is a host-state figure, not a config figure: the same guest read 16% and 49%
+  of resident memory on 2 MiB pages on two boots, depending on free memory and compaction before
+  the start (`docs/TESTING.md`). None of the numbers above were taken with the coverage checked,
+  which is one reason to trust the interleaved comparisons more than absolute values.
+- The `diskio` rows are not used for any conclusion: `rand4k_IOPS` moved between 5139 and 13650
+  across otherwise identical runs, which is the host page cache, not the guest.
+- The `vm-native-verify` sample quoted in the README and the docs page read 5012 Mops single-thread,
+  4% under the 5220-5330 the controlled runs above read on the same build. It was one run, taken
+  a minute after boot with the guest still settling. `vm-native-verify` is a pass/fail gate, and
+  its threshold is set for that; the cost claims here rest only on the interleaved runs.
+- The `#GP` intercept's individual cost has not been isolated; the three-column table above
+  bounds the whole five-patch set at "within run-to-run spread".