vfio-win10.md (10337B)
1 title: VFIO Install Notes 2 date: 2020-10-17 12:00 3 --- 4 You should first go look at [the Arch Wiki on it](https://wiki.archlinux.org/index.php/PCI%20passthrough%20via%20OVMF) or [Yuri Alek's guide on Single GPU passthrough](https://gitlab.com/YuriAlek/vfio) or [4chan's /g/ wiki on it](https://wiki.installgentoo.com/index.php/PCI_passthrough) as these assume prior knowledge. 5 6 # Prerequisites 7 ## UEFI Options 8 Enable VT-d and VT-x (or AMD equivalent) 9 10 ## Kernel Config 11 Enable KVM and VFIO 12 > you can set VFIO as builtin but as a module is more flexible 13 Also add `"iommu=pt intel_iommu=on"` to your kernel command line (or in CONFIG\_CMDLINE) 14 15 ### Current Options 16 ``` 17 ... 18 CONFIG_IOMMU_IOVA=y 19 CONFIG_IOMMU_API=y 20 CONFIG_IOMMU_SUPPORT=y 21 CONFIG_IOMMU_DEFAULT_PASSTHROUGH=y 22 # use the respective AMD options if using an AMD CPU 23 CONFIG_INTEL_IOMMU=y 24 CONFIG_INTEL_IOMMU_SVM=y 25 CONFIG_INTEL_IOMMU_DEFAULT_ON=y 26 CONFIG_INTEL_IOMMU_FLOPPY_WA=y 27 28 CONFIG_KVM_VFIO=y 29 CONFIG_VFIO_IOMMU_TYPE1=m 30 CONFIG_VFIO_VIRQFD=m 31 CONFIG_VFIO=m 32 CONFIG_VFIO_PCI=m 33 CONFIG_VFIO_PCI_VGA=y 34 CONFIG_VFIO_PCI_MMAP=y 35 CONFIG_VFIO_PCI_INTX=y 36 CONFIG_VFIO_PCI_IGD=y 37 CONFIG_VFIO_MDEV=m 38 CONFIG_VFIO_MDEV_DEVICE=m 39 ... 40 ``` 41 42 ## Packages Required 43 ``` 44 app-emulation/qemu (actual program) 45 sys-firmware/edk2-ovmf (UEFI firmware for Nvidia GPU) 46 media-sound/scream (audio) 47 looking-glass-client (compile from source if no package, or make your own) 48 ``` 49 50 `app-emulation/libvirt` can be used as well for easier configuration and autostart 51 but I have had problems with it: 52 - Service not starting properly, workaround is restarting service after it starts (Gentoo) 53 - Networks and domains not autostarting, workaround is starting them manually (CRUX) 54 55 ### Gentoo USE Flags 56 ``` 57 app-emulation/qemu gtk opengl sdl sdl-image usb # (spice, ssh, vhost-user-fs, virgl, and virtfs are optional I think) 58 media-libs/libsdl2 X gles opengl # for Looking Glass 59 ``` 60 note to self (2020-10-17): check how minimal you can make qemu to run vfio 61 62 # IOMMU 63 Run `dmesg | grep -E 'DMAR'` and see if `DMAR: IOMMU enabled` or something similar is in output 64 65 # QEMU Script 66 All code blocks in this section go in the qemu script file unless specified otherwise 67 68 ## Environment Variables 69 ```sh 70 IMG=/path/to/windows-image-file 71 VIRTIO=/path/to/virtio-iso 72 WINDOWS=/path/to/windows-install-iso 73 OVMF=/usr/share/edk2-ovmf/OVMF_CODE.fd 74 RAM=16G 75 ULIMIT=$(ulimit -l) 76 ULIMIT_TARGET=$(( $(echo $RAM | tr -d 'G')*1048576+100000 )) 77 78 GPU_VIDEO=01:00.0 79 GPU_AUDIO=01:00.1 80 VIDEOID="10de 13c0" 81 AUDIOID="10de 0fbb" 82 VIDEOBUSID="0000:${GPU_VIDEO}" 83 AUDIOBUSID="0000:${GPU_AUDIO}" 84 ``` 85 86 ## VFIO Detaching and Attaching 87 ```sh 88 vfio_on() { 89 # for nvidia card with proprietary drivers 90 rmmod nvidia_drm 91 rmmod nvidia_modeset 92 rmmod nvidia 93 94 # disable bumblebee service or use bbswitch to detach card if using bumblebee 95 96 modprobe vfio-pci 97 98 echo $VIDEOID > /sys/bus/pci/drivers/vfio-pci/new_id 99 echo $VIDEOBUSID > /sys/bus/pci/devices/$VIDEOBUSID/driver/unbind 100 echo $VIDEOBUSID > /sys/bus/pci/drivers/vfio-pci/bind 101 echo $VIDEOID > /sys/bus/pci/drivers/vfio-pci/remove_id 102 103 echo $AUDIOID > /sys/bus/pci/drivers/vfio-pci/new_id 104 echo $AUDIOBUSID > /sys/bus/pci/devices/$AUDIOBUSID/driver/unbind 105 echo $AUDIOBUSID > /sys/bus/pci/drivers/vfio-pci/bind 106 echo $AUDIOID > /sys/bus/pci/drivers/vfio-pci/remove_id 107 108 # add rest of gpu devices if they are in the same group (I think 4 devices in 1000 or 2000 series nvidia) 109 } 110 111 vfio_off() { 112 rmmod vfio_iommu_type1 113 rmmod vfio_pci 114 rmmod vfio_virqfd 115 rmmod vfio 116 117 modprobe nvidia 118 } 119 ``` 120 121 ## Networking 122 ```sh 123 net_on() { 124 ip tuntap add dev tap0 mode tap group kvm 125 ip link set dev tap0 up promisc on 126 ip addr add 0.0.0.0 dev tap0 127 128 ip link add br0 type bridge 129 ip link set br0 up 130 ip link set tap0 master br0 131 echo 0 > /sys/class/net/br0/bridge/stp_state 132 ip addr add 192.168.123.1/24 dev br0 133 134 sysctl net.ipv4.conf.tap0.proxy_arp=1 > /dev/null 135 sysctl net.ipv4.conf.enp0s31f6.proxy_arp=1 > /dev/null 136 sysctl net.ipv4.ip_forward=1 > /dev/null 137 138 iptables -t nat -A POSTROUTING -o enp0s31f6 -j MASQUERADE > /dev/null 139 iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT > /dev/null 140 iptables -A FORWARD -i br0 -o enp0s31f6 -j ACCEPT > /dev/null 141 } 142 143 net_off() { 144 sysctl net.ipv4.conf.tap0.proxy_arp=0 > /dev/null 145 sysctl net.ipv4.conf.enp0s31f6.proxy_arp=0 > /dev/null 146 sysctl net.ipv4.ip_forward=0 > /dev/null 147 148 ip link set dev br0 down 149 ip link del br0 150 151 ip link set dev tap0 down 152 ip tuntap del mode tap name tap0 153 } 154 ``` 155 156 Also add this to /etc/conf.d/net if using Gentoo ([source](https://wiki.gentoo.org/wiki/QEMU/Options#Network_bridge)) 157 > replace `enp0s31f6` with the host/master interface 158 ```sh 159 ... 160 tuntap_tap0="tap" 161 config_tap0="null" 162 bridge_br0="enp0s31f6 tap0" 163 164 config_br0="192.168.123.2 netmask 255.255.255.0" 165 routes_br0="default via 192.168.123.1" 166 bridge_forward_delay_br0=0 167 bridge_hello_time_br0=10 168 169 depend_br0() { 170 need net.enp0s31f6 171 need net.tap0 172 } 173 ... 174 ``` 175 176 ## Hugepages 177 ```sh 178 hugepages_on() { 179 PAGES=$(( $(echo $RAM | tr -d 'G') * 1048576 / 2048)) 180 mkdir -p /dev/hugepages 181 mount -t hugetlbfs hugetlbfs /dev/hugepages 182 echo $PAGES > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 183 } 184 185 hugepages_off() { 186 echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 187 umount /dev/hugepages 188 } 189 ``` 190 191 ## QEMU Command 192 ### Before installing guest OS (Windows 10 used as example) 193 ```sh 194 ulimit -l $ULIMIT_TARGET 195 196 qemu-system-x86_64 \ 197 -name 'vfio-vm' \ 198 -vga qxl \ 199 -nodefaults -enable-kvm -machine q35 \ 200 -m $RAM -mem-path /dev/hugepages \ 201 -cpu host,kvm=off,svm=off,topoext,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic,hv_vendor_id=novideobad43,hv_vpindex,hv_synic,hv_stimer,hv_frequencies \ 202 -smp 8,sockets=1,cores=4,threads=2 \ 203 -rtc clock=host,base=localtime \ 204 -boot menu=on -boot d \ 205 -nic tap,ifname=tap0,script=no,downscript=0,model=virtio-net-pci \ 206 -drive if=pflash,format=raw,readonly,file=$OVMF \ 207 -drive file="$VIRTIO",id=cd1,media=cdrom \ 208 -drive file="$WINDOWS",id=cd2,media=cdrom \ 209 -device virtio-scsi-pci,id=scsi0 \ 210 -device scsi-hd,bus=scsi0.0,drive=rootfs \ 211 -drive file="$IMG",id=rootfs,index=0,format=qcow2,media=disk,if=none 212 213 ulimit -l $ULIMIT 214 ``` 215 ### After installing guest OS 216 ```sh 217 ulimit -l $ULIMIT_TARGET 218 219 qemu-system-x86_64 \ 220 -name 'vfio-vm' \ 221 -vga none -nographic \ 222 -nodefaults -enable-kvm -machine q35 \ 223 -m $RAM -mem-path /dev/hugepages \ 224 -cpu host,kvm=off,svm=off,topoext,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic,hv_vendor_id=novideobad43,hv_vpindex,hv_synic,hv_stimer,hv_frequencies \ 225 -smp 8,sockets=1,cores=4,threads=2 \ 226 -rtc clock=host,base=localtime \ 227 -boot menu=on -boot c \ 228 -nic tap,ifname=tap0,script=no,downscript=0,model=virtio-net-pci \ 229 -device vfio-pci,host=$GPU_VIDEO,multifunction=on,x-vga=on \ 230 -device vfio-pci,host=$GPU_AUDIO \ 231 -device ivshmem-plain,memdev=ivshmem,bus=pcie.0 \ 232 -object memory-backend-file,id=ivshmem,share=on,mem-path=/dev/shm/looking-glass,size=32M \ 233 -device virtio-keyboard-pci \ 234 -device virtio-mouse-pci \ 235 -object input-linux,id=kbd0,evdev=/dev/input/by-id/usb-Corsair_Corsair_K70R_Gaming_Keyboard-if02-event-kbd,grab_all=on,repeat=on \ 236 -object input-linux,id=mouse0,evdev=/dev/input/by-id/usb-Logitech_Gaming_Mouse_G502_0E5F335C3236-event-mouse \ 237 -object input-linux,id=mouse1,evdev=/dev/input/by-id/usb-Logitech_Gaming_Mouse_G502_0E5F335C3236-if01-event-kbd,grab_all=on,repeat=on \ 238 -drive if=pflash,format=raw,readonly,file=$OVMF \ 239 -drive file="$VIRTIO",id=cd1,media=cdrom \ 240 -device virtio-scsi-pci,id=scsi0 \ 241 -device scsi-hd,bus=scsi0.0,drive=rootfs \ 242 -drive file="$IMG",id=rootfs,index=0,format=qcow2,media=disk,if=none 243 244 ulimit -l $ULIMIT 245 ``` 246 247 # Extra 248 ## Adding USB Devices 249 Get vendor and product id from `lsusb` and add them to your QEMU command arguments: 250 ```sh 251 -device qemu-xhci,id=xhci0 -device usb-host,bus=xhci0.0,vendorid=0x<yourvendorid>,productid=0x<yourproductid> 252 ``` 253 Example for my USB bluetooth receiver: 254 ``` 255 $ lsusb 256 ... 257 Bus 001 Device 004: ID 0b05:17cb ASUSTek Computer, Inc. Broadcom BCM20702A0 Bluetooth 258 ... 259 ``` 260 My vendorid is `0x0b05` and productid is `0x17cb`, so in QEMU it would be: 261 ```sh 262 -device qemu-xhci,id=<usb-bus-id> -device usb-host,bus=<usb-bus-id>.0,vendorid=0x0b05,productid=0x17cb 263 ``` 264 265 ## Set CPU Affinity 266 While libvirt makes this more simple, it appears we need a script/function to do it in bare QEMU 267 Borrowed from [here](https://null-src.com/posts/qemu-optimization/post.php#taskset) 268 > note: uses bash-isms so that's why I put it in a separate file 269 ```bash 270 #!/bin/bash 271 THREAD_LIST="0,4,1,5,2,6,3,7" 272 NAME="vfio-vm" 273 274 sleep 20 && 275 HOST_THREAD=0 276 # for each vCPU thread PID 277 for PID in $(pstree -pa $(pstree -pa $(pidof qemu-system-x86_64) | grep $NAME | awk -F',' '{print $2}' | awk '{print $1}') | grep CPU | pstree -pa $(pstree -pa $(pidof qemu-system-x86_64) | grep $NAME | cut -d',' -f2 | cut -d' ' -f1) | grep CPU | sort | awk -F',' '{print $2}') 278 do 279 let HOST_THREAD+=1 280 # set each vCPU thread PID to next host CPU thread in THREAD_LIST 281 echo "taskset -pc $(echo $THREAD_LIST | cut -d',' -f$HOST_THREAD) $PID" | sh 282 done 283 ``` 284 285 ## Additional Disk 286 You can add another disk by simply copying the arguments for adding the rootfs and slightly modifying 287 Example for a qcow2 image: 288 ``` 289 -device virtio-scsi-pci,id=<scsi-id> \ 290 -device scsi-hd,bus=<scsi-id>.0,drive=<drive-id> \ 291 -drive file=<location>,id=<drive-id>,index=0,format=qcow2,media=disk,if=none 292 ``` 293 294 ## No Drives During Installation 295 Make sure virtio driver is loaded: 296 - Click Load Driver 297 - Choose virtio-cd disc > amd64 > w10 and press enter 298 - Load Red Hat Virtio SCSI driver 299 300 ## Looking Glass Not Starting 301 Make sure no virtual display like QXL is loaded too (`-nographic -vga none` in QEMU) 302 303 ## JACK Support 304 To use JACK instead of Scream, you can use these QEMU arguments 305 ```sh 306 -audiodev jack,id=snd0,in.client-name=default,out.client-name=default,in.start-server=off,out.start-server=off,in.exact-name=on,out.exact-name=on,in.connect-ports=system,out.connect-ports=system,in.frequency=48000,out.frequency=48000,timer-period=2048,out.buffer-length=5120 \ 307 -device ich9-intel-hda \ 308 -device hda-output,audiodev=snd0 \ 309 ``` 310 You might need to change the timer-period and buffer-length if experiencing crackling. 311 Also you might have to change the controller (ich9-intel-hda) and codec (hda-output) to something else. 312 313 To list controller and codecs, run: 314 ```sh 315 qemu-system-x86_64 -device help | grep hda 316 ```