199 lines
10 KiB
Markdown
199 lines
10 KiB
Markdown
# From a plain Windows VM to the tuned, corrected one
|
|
|
|
`vm-native-setup` does the host side in one pass. This is the guest side, in the order that
|
|
works, plus the two host steps that have to happen between them. Everything here was run on a
|
|
Windows 11 guest; the order is the part that cost time to learn.
|
|
|
|
The shape of it:
|
|
|
|
```
|
|
base domain guest host guest again
|
|
----------- ----- ---- -----------
|
|
virtio disk, QXL, -> 1. hypervisor + VBS off -> 4. vm-native-setup -> 5. remove virtio
|
|
agents, 52:54:00 MAC 2. SSH + RDP on (disk to NVMe, devices, drivers + agents
|
|
3. stornvme boot-start firmware, identity) -> 6. score, verify
|
|
```
|
|
|
|
Steps 1 to 3 are done with the guest still on its virtio disk. Step 4 changes the disk bus, and
|
|
step 5 is only safe after that: remove `viostor` while the disk is still virtio and the next boot
|
|
is `INACCESSIBLE_BOOT_DEVICE`.
|
|
|
|
---
|
|
|
|
## 1. Turn off the guest's own hypervisor
|
|
|
|
A Windows guest that runs Hyper-V reports that truthfully, and no host-side setting changes it.
|
|
Virtualization-based security also costs 5 to 15 percent on CPU-bound work, so this is the one
|
|
step that makes the VM faster as well as quieter. As Administrator, in order:
|
|
|
|
```
|
|
bcdedit /set hypervisorlaunchtype off
|
|
bcdedit /set vsmlaunchtype off
|
|
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f
|
|
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard /v RequirePlatformSecurityFeatures /t REG_DWORD /d 0 /f
|
|
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\HypervisorEnforcedCodeIntegrity /v Enabled /t REG_DWORD /d 0 /f
|
|
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\CredentialGuard /v Enabled /t REG_DWORD /d 0 /f
|
|
reg add HKLM\SYSTEM\CurrentControlSet\Control\Lsa /v LsaCfgFlags /t REG_DWORD /d 0 /f
|
|
reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f
|
|
reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard /v HypervisorEnforcedCodeIntegrity /t REG_DWORD /d 0 /f
|
|
```
|
|
|
|
Then the optional features that bring a hypervisor with them. WSL is one of them and it is lost:
|
|
|
|
```powershell
|
|
Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All, Microsoft-Windows-Subsystem-Linux, VirtualMachinePlatform, HypervisorPlatform, Containers-DisposableClientVM -NoRestart
|
|
```
|
|
|
|
Reboot twice. Verify:
|
|
|
|
```powershell
|
|
(Get-CimInstance Win32_ComputerSystem).HypervisorPresent # False
|
|
(Get-CimInstance -Namespace root\Microsoft\Windows\DeviceGuard Win32_DeviceGuard).VirtualizationBasedSecurityStatus # 0
|
|
```
|
|
|
|
**Some installs will not let go.** On one Windows 11 install here the status stayed at 2 through
|
|
nine reboots with every one of the settings above applied. Memory Integrity had been on since
|
|
install and the policy is enforced from a place none of these keys reach. The two things that
|
|
worked: a clean install where VBS was never enabled, or the Core Isolation > Memory Integrity
|
|
toggle in Windows Security, then the commands again. Check the two values before anything else,
|
|
because every later step assumes they read `False` and `0`.
|
|
|
|
## 2. Two ways in that survive a dark console
|
|
|
|
The full profile removes the emulated display, and GPU passthrough hands the host's screen to the
|
|
guest. Both need a way in that does not depend on a console. As Administrator:
|
|
|
|
```powershell
|
|
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
|
|
Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
|
|
Set-Service sshd -StartupType Automatic; Start-Service sshd
|
|
New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
|
|
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server" /v fDenyTSConnections /t REG_DWORD /d 0 /f
|
|
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"
|
|
```
|
|
|
|
Put your public key in `C:\ProgramData\ssh\administrators_authorized_keys` for an Administrator
|
|
account. `vm-native-verify` uses that key.
|
|
|
|
SSH into the guest fails with `Corrupted MAC on input` until the host has the e1000e offload rule
|
|
from the `vfio-native` package. The emulated NIC's TX offloads corrupt integrity-checked traffic on
|
|
the host side of the tap; SMB tolerates it, SSH does not. The package installs a udev rule that
|
|
turns the offloads off on every libvirt tap as it appears, and `vm-native-setup` says so if it is
|
|
missing.
|
|
|
|
## 3. Make the NVMe driver boot-critical
|
|
|
|
The disk is about to move from virtio to emulated NVMe, and Windows only loads boot-start drivers
|
|
before it can read the disk. `stornvme` is inbox but not always boot-start on an install that
|
|
never saw an NVMe disk. Before the move:
|
|
|
|
```
|
|
reg add HKLM\SYSTEM\CurrentControlSet\Services\stornvme /v Start /t REG_DWORD /d 0 /f
|
|
reg add HKLM\SYSTEM\CurrentControlSet\Services\storahci /v Start /t REG_DWORD /d 0 /f
|
|
reg delete HKLM\SYSTEM\CurrentControlSet\Services\stornvme\StartOverride /f
|
|
reg delete HKLM\SYSTEM\CurrentControlSet\Services\storahci\StartOverride /f
|
|
```
|
|
|
|
The `reg delete` lines may say the key does not exist, which is fine. Skip this step and the
|
|
first NVMe boot ends at Windows Boot Manager with `0xc0000225`, "a required device isn't connected
|
|
or can't be accessed". That is exactly what a set-aside copy of the daily VM's disk did here.
|
|
|
|
Shut the guest down.
|
|
|
|
## 4. The host pass
|
|
|
|
```sh
|
|
vm-native-setup -d win11 -p full -r
|
|
```
|
|
|
|
It moves every disk to emulated NVMe with a serial, replaces the virtio device set (balloon,
|
|
RNG, serial channel, agent channel, virtiofs, virtio NIC and inputs) with what a real board has,
|
|
removes the emulated display for the full level, wires Secure Boot with a key store it generates,
|
|
writes the SMBIOS and ACPI identity, pins the vCPUs, and points the domain at the patched QEMU.
|
|
`-r` gives this deployment its own serials, MAC and memory module, which matters because every
|
|
installation of a tool that ships fixed identity strings shares one fingerprint. The generated
|
|
values live in `~/.local/share/vfio-native/<domain>/identity.env` and stay put until you pass `-r`
|
|
again.
|
|
|
|
Two things it asks or warns about:
|
|
|
|
- **USB passthrough.** It lists your devices and which controller each sits behind. `auto` passes
|
|
keyboard and mouse: a whole controller when only they sit on it and its IOMMU group is clean,
|
|
the individual devices otherwise. The host loses whatever is passed for as long as the guest
|
|
runs, so have SSH working first.
|
|
- **The firmware store.** Enrolling Secure Boot keys means the domain's EFI variable store is
|
|
recreated from the new template. Boot entries come back on their own. If BitLocker is on in the
|
|
guest, suspend it first or the next boot asks for the recovery key.
|
|
|
|
Before the first boot, if the host has less free memory than the guest's RAM, free and compact
|
|
it so the guest lands on transparent hugepages; `vm-native-setup` prints the two commands when it
|
|
applies. The NIC stays `e1000e`, so the network survives the driver removal in the next step. Do not use
|
|
virtiofs for host files: it is a virtio device the scanner names, and its shared memory backing
|
|
blocks transparent hugepages for the whole guest. Share over SMB on the e1000e link instead.
|
|
|
|
## 5. Remove the virtio drivers and the agents
|
|
|
|
Boot the guest. It is on NVMe now, so `viostor` and `vioscsi` can go. In this order, as
|
|
Administrator:
|
|
|
|
1. Uninstall `virtio-win-guest-tools` from Apps, or its cached installer with `/uninstall /quiet`.
|
|
Then `msiexec /x` whatever it leaves behind: the QEMU guest agent, the Spice agent, the Red Hat
|
|
QXL driver, the virtio-win driver installer.
|
|
2. Remove the Red Hat packages from the driver store:
|
|
|
|
```
|
|
pnputil /enum-drivers
|
|
pnputil /delete-driver oemNN.inf /uninstall /force
|
|
```
|
|
|
|
one per `Red Hat` or `virtio` entry the first command lists.
|
|
3. Delete the service keys they leave behind:
|
|
|
|
```
|
|
for %s in (viostor vioscsi netkvm vioser BalloonService VirtioFsSvc pvpanic vioinput viorng qemu-ga spice-agent) do reg delete HKLM\SYSTEM\CurrentControlSet\Services\%s /f
|
|
```
|
|
|
|
Reboot. Nothing in the guest now names the emulator.
|
|
|
|
## 6. Score and verify
|
|
|
|
From the host, with the guest up for a minute:
|
|
|
|
```sh
|
|
echo 'AMD Ryzen 7 7700X 8-Core Processor' | sudo tee /sys/module/kvm_amd/parameters/brand_string
|
|
echo Y | sudo tee /sys/module/kvm_amd/parameters/cpuid_passthrough
|
|
vm-native-verify
|
|
```
|
|
|
|
`vm-native-setup` printed those two lines with your SKU, or `sudo vm-native-cpuid enable <domain>` does the same on/off automatically around the guest (off for the cold boot, on once its network is up). The switch is host-wide: it applies to
|
|
every pinned guest at once, and a cold boot of any of them while it is on hits the enumeration
|
|
race, so with several guests turn it off before any boots and on when they are all up. Then the
|
|
scanner, from the console session, as `docs/TESTING.md` describes. The reference guest reads 1/85.
|
|
|
|
## 7. GPU and the rest of the desk
|
|
|
|
```sh
|
|
sudo vm-native-gpu --single win11 0000:03:00.0 /path/to/vbios.rom
|
|
```
|
|
|
|
writes every function of the card into the domain, adds the vBIOS as the ROM file if you give
|
|
one, removes the emulated display, and installs the hook that frees the card when the guest starts
|
|
and gives it back when it stops. The host has no screen while the guest runs. `--dual` is the same
|
|
without the hook, for a second card the host does not use. USB devices were handled in step 4;
|
|
re-run `vm-native-setup -u <spec>` to change them.
|
|
|
|
---
|
|
|
|
## What is not fixed by any of this
|
|
|
|
- **`GPU_CAPABILITIES`** needs the real GPU. Nothing emulated passes it.
|
|
- **A guest that has run its own hypervisor.** If step 1's two values will not read `False` and
|
|
`0`, no host setting rescues it. Reinstall.
|
|
- **Windows activation.** `-r` changes the board serial and the MAC. A digital licence tied to
|
|
the previous hardware hash may want re-activation. Run `-r` once, early, not per boot.
|
|
- **The EDID.** The emulated monitor's serial is compiled into the `07-edid` QEMU patch and `-r`
|
|
does not touch it. It only exists while the guest still has an emulated display; the full level
|
|
has none.
|
|
- **Memory module serials.** QEMU takes one string set for every DIMM, so all modules report the
|
|
same serial. Real boards do not, and nothing scored here reads it.
|