vfio-native/docs/GUEST-SETUP.md

From a plain Windows VM to the tuned, corrected one

vm-native-setup does the host side in one pass. This is the guest side, in the order that works, plus the two host steps that have to happen between them. Everything here was run on a Windows 11 guest; the order is the part that cost time to learn.

The shape of it:

  base domain              guest                       host                      guest again
  -----------              -----                       ----                      -----------
  virtio disk, QXL,   ->   1. hypervisor + VBS off  -> 4. vm-native-setup    ->  5. remove virtio
  agents, 52:54:00 MAC     2. SSH + RDP on             (disk to NVMe, devices,     drivers + agents
                           3. stornvme boot-start       firmware, identity)    ->  6. score, verify

Steps 1 to 3 are done with the guest still on its virtio disk. Step 4 changes the disk bus, and step 5 is only safe after that: remove viostor while the disk is still virtio and the next boot is INACCESSIBLE_BOOT_DEVICE.

---

1. Turn off the guest's own hypervisor

A Windows guest that runs Hyper-V reports that truthfully, and no host-side setting changes it. Virtualization-based security also costs 5 to 15 percent on CPU-bound work, so this is the one step that makes the VM faster as well as quieter. As Administrator, in order:

bcdedit /set hypervisorlaunchtype off
bcdedit /set vsmlaunchtype off
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard /v RequirePlatformSecurityFeatures /t REG_DWORD /d 0 /f
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\HypervisorEnforcedCodeIntegrity /v Enabled /t REG_DWORD /d 0 /f
reg add HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\CredentialGuard /v Enabled /t REG_DWORD /d 0 /f
reg add HKLM\SYSTEM\CurrentControlSet\Control\Lsa /v LsaCfgFlags /t REG_DWORD /d 0 /f
reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f
reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\DeviceGuard /v HypervisorEnforcedCodeIntegrity /t REG_DWORD /d 0 /f

Then the optional features that bring a hypervisor with them. WSL is one of them and it is lost:

Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-All, Microsoft-Windows-Subsystem-Linux, VirtualMachinePlatform, HypervisorPlatform, Containers-DisposableClientVM -NoRestart

Reboot twice. Verify:

(Get-CimInstance Win32_ComputerSystem).HypervisorPresent            # False
(Get-CimInstance -Namespace root\Microsoft\Windows\DeviceGuard Win32_DeviceGuard).VirtualizationBasedSecurityStatus   # 0

Some installs will not let go. On one Windows 11 install here the status stayed at 2 through nine reboots with every one of the settings above applied. Memory Integrity had been on since install and the policy is enforced from a place none of these keys reach. The two things that worked: a clean install where VBS was never enabled, or the Core Isolation > Memory Integrity toggle in Windows Security, then the commands again. Check the two values before anything else, because every later step assumes they read False and 0.

2. Two ways in that survive a dark console

The full profile removes the emulated display, and GPU passthrough hands the host's screen to the guest. Both need a way in that does not depend on a console. As Administrator:

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
Set-Service sshd -StartupType Automatic; Start-Service sshd
New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server" /v fDenyTSConnections /t REG_DWORD /d 0 /f
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"

Put your public key in C:\ProgramData\ssh\administrators_authorized_keys for an Administrator account. vm-native-verify uses that key.

3. Make the NVMe driver boot-critical

The disk is about to move from virtio to emulated NVMe, and Windows only loads boot-start drivers before it can read the disk. stornvme is inbox but not always boot-start on an install that never saw an NVMe disk. Before the move:

reg add HKLM\SYSTEM\CurrentControlSet\Services\stornvme /v Start /t REG_DWORD /d 0 /f
reg add HKLM\SYSTEM\CurrentControlSet\Services\storahci /v Start /t REG_DWORD /d 0 /f
reg delete HKLM\SYSTEM\CurrentControlSet\Services\stornvme\StartOverride /f
reg delete HKLM\SYSTEM\CurrentControlSet\Services\storahci\StartOverride /f

The reg delete lines may say the key does not exist, which is fine. Skip this step and the first NVMe boot ends at Windows Boot Manager with 0xc0000225, "a required device isn't connected or can't be accessed". That is exactly what a set-aside copy of the daily VM's disk did here.

Shut the guest down.

4. The host pass

vm-native-setup -d win11 -p full -r

It moves every disk to emulated NVMe with a serial, replaces the virtio device set (balloon, RNG, serial channel, agent channel, virtiofs, virtio NIC and inputs) with what a real board has, removes the emulated display for the full level, wires Secure Boot with a key store it generates, writes the SMBIOS and ACPI identity, pins the vCPUs, and points the domain at the patched QEMU. -r gives this deployment its own serials, MAC and memory module, which matters because every installation of a tool that ships fixed identity strings shares one fingerprint. The generated values live in ~/.local/share/vfio-native/<domain>/identity.env and stay put until you pass -r again.

Two things it asks or warns about:

• USB passthrough. It lists your devices and which controller each sits behind. auto passes keyboard and mouse: a whole controller when only they sit on it and its IOMMU group is clean, the individual devices otherwise. The host loses whatever is passed for as long as the guest runs, so have SSH working first.
• The firmware store. Enrolling Secure Boot keys means the domain's EFI variable store is recreated from the new template. Boot entries come back on their own. If BitLocker is on in the guest, suspend it first or the next boot asks for the recovery key.

Before the first boot, if the host has less free memory than the guest's RAM, free and compact it so the guest lands on transparent hugepages; vm-native-setup prints the two commands when it applies. The NIC stays igb, so the network survives the driver removal in the next step. Do not use virtiofs for host files: it is a virtio device the scanner names, and its shared memory backing blocks transparent hugepages for the whole guest. Share over SMB on the igb link instead.

The interface asks for MTU 9000, and the package's hook turns GSO and GRO on for the tap. Together they take the inbound link from 2.9 to 14 Gbit/s; neither does anything alone. The libvirt network needs <mtu size='9000'/> too, and the guest needs Jumbo Packet 9014 with its interface MTU at 9000 - setting the adapter property alone leaves the IP MTU at 1500 and gains nothing.

5. Remove the virtio drivers and the agents

Boot the guest. It is on NVMe now, so viostor and vioscsi can go. In this order, as Administrator:

1. Uninstall virtio-win-guest-tools from Apps, or its cached installer with /uninstall /quiet. Then msiexec /x whatever it leaves behind: the QEMU guest agent, the Spice agent, the Red Hat QXL driver, the virtio-win driver installer.

2. Remove the Red Hat packages from the driver store:

   pnputil /enum-drivers
   pnputil /delete-driver oemNN.inf /uninstall /force
   

   one per Red Hat or virtio entry the first command lists.

3. Delete the service keys they leave behind:

   for %s in (viostor vioscsi netkvm vioser BalloonService VirtioFsSvc pvpanic vioinput viorng qemu-ga spice-agent) do reg delete HKLM\SYSTEM\CurrentControlSet\Services\%s /f
   

Reboot. Nothing in the guest now names the emulator.

6. Score and verify

From the host, with the guest up for a minute:

echo 'AMD Ryzen 7 7700X 8-Core Processor' | sudo tee /sys/module/kvm_amd/parameters/brand_string
echo Y | sudo tee /sys/module/kvm_amd/parameters/cpuid_passthrough
vm-native-verify

vm-native-setup printed those two lines with your SKU, or sudo vm-native-cpuid enable <domain> does the same on/off automatically around the guest (off for the cold boot, on once its network is up). The switch is host-wide: it applies to every pinned guest at once, and a cold boot of any of them while it is on hits the enumeration race, so with several guests turn it off before any boots and on when they are all up. Then the scanner, from the console session, as docs/TESTING.md describes. The reference guest reads 1/85.

7. GPU and the rest of the desk

sudo vm-native-gpu --single win11 0000:03:00.0 /path/to/vbios.rom

writes every function of the card into the domain, adds the vBIOS as the ROM file if you give one, removes the emulated display, and installs the hook that frees the card when the guest starts and gives it back when it stops. The host has no screen while the guest runs. --dual is the same without the hook, for a second card the host does not use. USB devices were handled in step 4; re-run vm-native-setup -u <spec> to change them.

---

What is not fixed by any of this

• GPU_CAPABILITIES needs the real GPU. Nothing emulated passes it.
• A guest that has run its own hypervisor. If step 1's two values will not read False and 0, no host setting rescues it. Reinstall.
• Windows activation. -r changes the board serial and the MAC. A digital licence tied to the previous hardware hash may want re-activation. Run -r once, early, not per boot.
• The EDID. The emulated monitor's serial is compiled into the 07-edid QEMU patch and -r does not touch it. It only exists while the guest still has an emulated display; the full level has none.
• Memory module serials. QEMU takes one string set for every DIMM, so all modules report the same serial. Real boards do not, and nothing scored here reads it.