docs: readme, analysis, results, testing and guest setup
This commit is contained in:
198
docs/GUEST-SETUP.md
Normal file
198
docs/GUEST-SETUP.md
Normal file
@@ -0,0 +1,198 @@
|
||||
# From a plain Windows VM to the tuned, corrected one
|
||||
|
||||
`vm-native-setup` does the host side in one pass. This is the guest side, in the order that
|
||||
works, plus the two host steps that have to happen between them. Everything here was run on a
|
||||
Windows 11 guest; the order is the part that cost time to learn.
|
||||
|
||||
The shape of it:
|
||||
|
||||
```
|
||||
base domain guest host guest again
|
||||
----------- ----- ---- -----------
|
||||
virtio disk, QXL, -> 1. hypervisor + VBS off -> 4. vm-native-setup -> 5. remove virtio
|
||||
agents, 52:54:00 MAC 2. SSH + RDP on (disk to NVMe, devices, drivers + agents
|
||||
3. stornvme boot-start firmware, identity) -> 6. score, verify
|
||||
```
|
||||
|
||||
Steps 1 to 3 are done with the guest still on its virtio disk. Step 4 changes the disk bus, and
|
||||
step 5 is only safe after that: remove `viostor` while the disk is still virtio and the next boot
|
||||
is `INACCESSIBLE_BOOT_DEVICE`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Turn off the guest's own hypervisor
|
||||
|
||||
A Windows guest that runs Hyper-V reports that truthfully, and no host-side setting changes it.
|
||||
Virtualization-based security also costs 5 to 15 percent on CPU-bound work, so this is the one
|
||||
step that makes the VM faster as well as quieter. As Administrator, in order:
|
||||
|
||||
```
|
||||
bcdedit /set hypervisorlaunchtype off
|
||||
bcdedit /set vsmlaunchtype off
|
||||
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f
|
||||
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard /v RequirePlatformSecurityFeatures /t REG_DWORD /d 0 /f
|
||||
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\HypervisorEnforcedCodeIntegrity /v Enabled /t REG_DWORD /d 0 /f
|
||||
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\CredentialGuard /v Enabled /t REG_DWORD /d 0 /f
|
||||
reg add HKLM\SYSTEM\CurrentControlSet\Control\Lsa /v LsaCfgFlags /t REG_DWORD /d 0 /f
|
||||
reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f
|
||||
reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard /v HypervisorEnforcedCodeIntegrity /t REG_DWORD /d 0 /f
|
||||
```
|
||||
|
||||
Then the optional features that bring a hypervisor with them. WSL is one of them and it is lost:
|
||||
|
||||
```powershell
|
||||
Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All, Microsoft-Windows-Subsystem-Linux, VirtualMachinePlatform, HypervisorPlatform, Containers-DisposableClientVM -NoRestart
|
||||
```
|
||||
|
||||
Reboot twice. Verify:
|
||||
|
||||
```powershell
|
||||
(Get-CimInstance Win32_ComputerSystem).HypervisorPresent # False
|
||||
(Get-CimInstance -Namespace root\Microsoft\Windows\DeviceGuard Win32_DeviceGuard).VirtualizationBasedSecurityStatus # 0
|
||||
```
|
||||
|
||||
**Some installs will not let go.** On one Windows 11 install here the status stayed at 2 through
|
||||
nine reboots with every one of the settings above applied. Memory Integrity had been on since
|
||||
install and the policy is enforced from a place none of these keys reach. The two things that
|
||||
worked: a clean install where VBS was never enabled, or the Core Isolation > Memory Integrity
|
||||
toggle in Windows Security, then the commands again. Check the two values before anything else,
|
||||
because every later step assumes they read `False` and `0`.
|
||||
|
||||
## 2. Two ways in that survive a dark console
|
||||
|
||||
The full profile removes the emulated display, and GPU passthrough hands the host's screen to the
|
||||
guest. Both need a way in that does not depend on a console. As Administrator:
|
||||
|
||||
```powershell
|
||||
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
|
||||
Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
|
||||
Set-Service sshd -StartupType Automatic; Start-Service sshd
|
||||
New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
|
||||
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server" /v fDenyTSConnections /t REG_DWORD /d 0 /f
|
||||
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"
|
||||
```
|
||||
|
||||
Put your public key in `C:\ProgramData\ssh\administrators_authorized_keys` for an Administrator
|
||||
account. `vm-native-verify` uses that key.
|
||||
|
||||
SSH into the guest fails with `Corrupted MAC on input` until the host has the e1000e offload rule
|
||||
from the `vfio-native` package. The emulated NIC's TX offloads corrupt integrity-checked traffic on
|
||||
the host side of the tap; SMB tolerates it, SSH does not. The package installs a udev rule that
|
||||
turns the offloads off on every libvirt tap as it appears, and `vm-native-setup` says so if it is
|
||||
missing.
|
||||
|
||||
## 3. Make the NVMe driver boot-critical
|
||||
|
||||
The disk is about to move from virtio to emulated NVMe, and Windows only loads boot-start drivers
|
||||
before it can read the disk. `stornvme` is inbox but not always boot-start on an install that
|
||||
never saw an NVMe disk. Before the move:
|
||||
|
||||
```
|
||||
reg add HKLM\SYSTEM\CurrentControlSet\Services\stornvme /v Start /t REG_DWORD /d 0 /f
|
||||
reg add HKLM\SYSTEM\CurrentControlSet\Services\storahci /v Start /t REG_DWORD /d 0 /f
|
||||
reg delete HKLM\SYSTEM\CurrentControlSet\Services\stornvme\StartOverride /f
|
||||
reg delete HKLM\SYSTEM\CurrentControlSet\Services\storahci\StartOverride /f
|
||||
```
|
||||
|
||||
The `reg delete` lines may say the key does not exist, which is fine. Skip this step and the
|
||||
first NVMe boot ends at Windows Boot Manager with `0xc0000225`, "a required device isn't connected
|
||||
or can't be accessed". That is exactly what a set-aside copy of the daily VM's disk did here.
|
||||
|
||||
Shut the guest down.
|
||||
|
||||
## 4. The host pass
|
||||
|
||||
```sh
|
||||
vm-native-setup -d win11 -p full -r
|
||||
```
|
||||
|
||||
It moves every disk to emulated NVMe with a serial, replaces the virtio device set (balloon,
|
||||
RNG, serial channel, agent channel, virtiofs, virtio NIC and inputs) with what a real board has,
|
||||
removes the emulated display for the full level, wires Secure Boot with a key store it generates,
|
||||
writes the SMBIOS and ACPI identity, pins the vCPUs, and points the domain at the patched QEMU.
|
||||
`-r` gives this deployment its own serials, MAC and memory module, which matters because every
|
||||
installation of a tool that ships fixed identity strings shares one fingerprint. The generated
|
||||
values live in `~/.local/share/vfio-native/<domain>/identity.env` and stay put until you pass `-r`
|
||||
again.
|
||||
|
||||
Two things it asks or warns about:
|
||||
|
||||
- **USB passthrough.** It lists your devices and which controller each sits behind. `auto` passes
|
||||
keyboard and mouse: a whole controller when only they sit on it and its IOMMU group is clean,
|
||||
the individual devices otherwise. The host loses whatever is passed for as long as the guest
|
||||
runs, so have SSH working first.
|
||||
- **The firmware store.** Enrolling Secure Boot keys means the domain's EFI variable store is
|
||||
recreated from the new template. Boot entries come back on their own. If BitLocker is on in the
|
||||
guest, suspend it first or the next boot asks for the recovery key.
|
||||
|
||||
Before the first boot, if the host has less free memory than the guest's RAM, free and compact
|
||||
it so the guest lands on transparent hugepages; `vm-native-setup` prints the two commands when it
|
||||
applies. The NIC stays `e1000e`, so the network survives the driver removal in the next step. Do not use
|
||||
virtiofs for host files: it is a virtio device the scanner names, and its shared memory backing
|
||||
blocks transparent hugepages for the whole guest. Share over SMB on the e1000e link instead.
|
||||
|
||||
## 5. Remove the virtio drivers and the agents
|
||||
|
||||
Boot the guest. It is on NVMe now, so `viostor` and `vioscsi` can go. In this order, as
|
||||
Administrator:
|
||||
|
||||
1. Uninstall `virtio-win-guest-tools` from Apps, or its cached installer with `/uninstall /quiet`.
|
||||
Then `msiexec /x` whatever it leaves behind: the QEMU guest agent, the Spice agent, the Red Hat
|
||||
QXL driver, the virtio-win driver installer.
|
||||
2. Remove the Red Hat packages from the driver store:
|
||||
|
||||
```
|
||||
pnputil /enum-drivers
|
||||
pnputil /delete-driver oemNN.inf /uninstall /force
|
||||
```
|
||||
|
||||
one per `Red Hat` or `virtio` entry the first command lists.
|
||||
3. Delete the service keys they leave behind:
|
||||
|
||||
```
|
||||
for %s in (viostor vioscsi netkvm vioser BalloonService VirtioFsSvc pvpanic vioinput viorng qemu-ga spice-agent) do reg delete HKLM\SYSTEM\CurrentControlSet\Services\%s /f
|
||||
```
|
||||
|
||||
Reboot. Nothing in the guest now names the emulator.
|
||||
|
||||
## 6. Score and verify
|
||||
|
||||
From the host, with the guest up for a minute:
|
||||
|
||||
```sh
|
||||
echo 'AMD Ryzen 7 7700X 8-Core Processor' | sudo tee /sys/module/kvm_amd/parameters/brand_string
|
||||
echo Y | sudo tee /sys/module/kvm_amd/parameters/cpuid_passthrough
|
||||
vm-native-verify
|
||||
```
|
||||
|
||||
`vm-native-setup` printed those two lines with your SKU. The switch is host-wide: it applies to
|
||||
every pinned guest at once, and a cold boot of any of them while it is on hits the enumeration
|
||||
race, so with several guests turn it off before any boots and on when they are all up. Then the
|
||||
scanner, from the console session, as `docs/TESTING.md` describes. The reference guest reads 1/85.
|
||||
|
||||
## 7. GPU and the rest of the desk
|
||||
|
||||
```sh
|
||||
sudo vm-native-gpu --single win11 0000:03:00.0 /path/to/vbios.rom
|
||||
```
|
||||
|
||||
writes every function of the card into the domain, adds the vBIOS as the ROM file if you give
|
||||
one, removes the emulated display, and installs the hook that frees the card when the guest starts
|
||||
and gives it back when it stops. The host has no screen while the guest runs. `--dual` is the same
|
||||
without the hook, for a second card the host does not use. USB devices were handled in step 4;
|
||||
re-run `vm-native-setup -u <spec>` to change them.
|
||||
|
||||
---
|
||||
|
||||
## What is not fixed by any of this
|
||||
|
||||
- **`GPU_CAPABILITIES`** needs the real GPU. Nothing emulated passes it.
|
||||
- **A guest that has run its own hypervisor.** If step 1's two values will not read `False` and
|
||||
`0`, no host setting rescues it. Reinstall.
|
||||
- **Windows activation.** `-r` changes the board serial and the MAC. A digital licence tied to
|
||||
the previous hardware hash may want re-activation. Run `-r` once, early, not per boot.
|
||||
- **The EDID.** The emulated monitor's serial is compiled into the `07-edid` QEMU patch and `-r`
|
||||
does not touch it. It only exists while the guest still has an emulated display; the full level
|
||||
has none.
|
||||
- **Memory module serials.** QEMU takes one string set for every DIMM, so all modules report the
|
||||
same serial. Real boards do not, and nothing scored here reads it.
|
||||
Reference in New Issue
Block a user