In one command it kills X, frees the GPU from drivers and console, detaches the GPU from the host, starts the VM with the GPU, waits until the VM is off, reattaches the GPU to the host and starts lightdm.
## What you need
* An IOMMU enabled motherboard. Check your motherboard manual.
* CPU support for AMD-v/VT-x and AMD-Vi/VT-d (AMD/Intel). And virtualization support enabled on BIOS.
* One GPU that supports UEFI and its BIOS. All GPUs from 2012 and later should support this. If the GPU does not support UEFI you may be able to make it work, but you wont see anything in the screen until the drivers inside Windows kick in.
* An IOMMU enabled motherboard. Check your motherboard manual for an option in BIOS to enable IOMMU.
* CPU support for AMD-v/VT-x and AMD-Vi/VT-d (AMD/Intel).
* Virtualization support enabled on BIOS; check your motherboard manual if you need help.
* One GPU that supports UEFI and its BIOS. All GPUs from 2012 and later should support this, some may have issues. If the GPU does not support UEFI you may be able to make it work, but you won't see anything in the screen until the drivers inside Windows kick in.
* QEMU, OVMF UEFI and VIRTIO drivers for Windows.
* [Optional] HDD only for Windows
* [Recommended] Another computer to login remotely with `ssh` for convenience, at least until you have everything working.
@ -49,25 +50,25 @@ In one command it kills X, frees the GPU from drivers and console, detaches the
```
## vBIOS
I experienced some weird things when doing this on the display, like a corruption of the image, but seems to work fine. If you encounter anything, a reboot solved my problems.
I experienced some weird things when doing this on the display, like a corruption of the image, it may be my GPU. If you encounter anything, a reboot solved my problems.
### Method 1 - Linux
It did not work for me, the ROM is 59KiB and it should be around 162KiB. It may work for you.
1. Execute `scripts/iommu.sh`as **root**to get the BUS ID for the GPU. Looks like `0000:06:00.0`.
1. Execute `scripts/iommu.sh` to get the BUS ID for the GPU. Looks like `0000:06:00.0`.
2. Edit `scripts/extract-vbios-linux.sh` to your convenience. Change `videobusid=`. [Optional] Change also the location where the vBIOS will be save `VBIOS=`.
3. Make the script executable with `chmod +x scripts/extract-vbios-linux.sh`
4. Link the service to systemd: `ln -s scripts/qemu@.service /usr/lib/systemd/system/`
5. Execute the systemd unit with `sudo systemctl start qemu@extract-vbios-linux.service`. You can also do it over `ssh`. The extracted ROM will be in the root directory `/root/vBIOS.rom`
3. Make the script executable with `chmod +x scripts/extract-vbios-linux.sh`.
4. Link the service to systemd: `ln -s scripts/qemu@.service /usr/lib/systemd/system/`.
5. Execute the systemd unit with `sudo systemctl start qemu@extract-vbios-linux.service`. You can also do it over `ssh`. The screen will turn dark for a while. The extracted ROM will be in the root directory `/root/vBIOS.rom`
From here you are alone, I don't know how to proceed. Maybe you need to edit, maybe don't.
### Method 2 - nvflash in Linux
1. Download nvflash https://www.techpowerup.com/download/nvidia-nvflash/. Do not install from AUR; the package it's broken.
2. Unzip it as `/root/nvflash_linux` with `# unzip nvflash_5.414.0_linux.zip -d /root/`
3. Execute `scripts/iommu.sh`as **root**to get the BUS ID for the GPU. Looks like `0000:06:00.0`.
4. Edit `scripts/extract-vbios-nvflash.sh`. Change `videobusid` with your GPU; `NVFLASH=` if you changed the location of the executable; and `VBIOS=` if you want the ROM in other place.
5. Link the service to systemd: `ln -s scripts/qemu@.service /usr/lib/systemd/system/`
6. Execute the systemd unit with `sudo systemctl start qemu@extract-vbios-nvflash.service`. You can also do it over `ssh`. The extracted ROM will be in the root directory `/root/vBIOS.rom`
2. Unzip it as `/root/nvflash_linux` with `# unzip nvflash_5.414.0_linux.zip -d /root/`.
3. Execute `scripts/iommu.sh` to get the BUS ID for the GPU. Looks like `0000:06:00.0`.
4. Edit `scripts/extract-vbios-nvflash.sh`. Change the variables `videobusid` with your GPU BUS ID; `NVFLASH` if you changed the location of the executable; and `VBIOS` if you want the ROM in other path.
5. Link the service to systemd: `ln -s scripts/qemu@.service /usr/lib/systemd/system/`.
6. Execute the systemd unit with `sudo systemctl start qemu@extract-vbios-nvflash.service`. You can also do it over `ssh`. The screen will turn dark for a while. The extracted ROM will be in the root directory `/root/vBIOS.rom`
7. [Edit the vBIOS](#edit-the-vbios)
### Method 3 - Windows
@ -86,12 +87,12 @@ From here you are alone, I don't know how to proceed. Maybe you need to edit, ma
$ git clone https://gitlab.com/YuriAlek/vfio.git
```
2. [Optional] [Download virtio drivers](https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers). If you do not, modify the `windows.sh` script.
2. [Optional] [Download virtio drivers](https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers). If you do not, modify `scripts/windows.sh`.
4. Get the iommu groups needed for the VM (GPU, GPU audio and USB controller)
4. Get the iommu groups needed for the VM (GPU, GPU audio and USB controller). They look like `06:00.0`
```
$ chmod +x scripts/iommu.sh
$ scripts/iommu.sh
@ -108,18 +109,21 @@ IOMMU group 18
08:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
```
5. [Optional] Create the image for the VM. Only if not using a physical hard drive.
5. [Optional] Create the image for the VM. Only if not using a physical hard drive. You can edit the path, the size and the format. Check `man qemu-img` for more information.
```bash
$ qemu-img create -f raw windows.raw 60G
$ qemu-img create -f raw /path/to/image/windows.raw 60G
```
6. Edit the script `windows-install.sh`**and**`windows.sh` to convenience. Things you may have to edit:
1. PCI devices
2. User
3. Location of HDD, ISO, vBIOS and OVMF image
4. The Desktop Environment, Display Manager, Window Manager, etc.
5. QEMU options like RAM and CPU
6. Kernel modules
6. Edit the script `windows-install.sh`**and**`windows.sh` to convenience. Variables you may have to edit:
1. PCI devices. `IOMMU_GPU`; `IOMMU_USB`.
2. User.
3. Location of HDD/IMG, ISO, vBIOS and OVMF image.
4. The Desktop Environment, Display Manager, Window Manager, etc. `lightdm`/`i3`.
5. QEMU options like RAM and CPU cores.
6. Kernel modules.
7. Other things like add a command to kill PulseAudio `pulseaudio -k` and another, at the end of the script, to start it again `pulseaudio --start`.
8. The network options `-device virtio-net-pci,netdev=n1 -netdev user,id=n1`.
9. Swap virtio for sata as the HDD interface.
7. Start the VM
```
@ -163,7 +167,7 @@ $ macos
```
### For MacOS
[Check this amazing guide](https://github.com/kholia/OSX-KVM) for creating the MacOS install image and Clover.
[Check this amazing guide for creating the MacOS install image and Clover](https://github.com/kholia/OSX-KVM).
## Known problems
### Race condition
@ -179,25 +183,27 @@ Sometimes the QEMU command will just fail and the command continues and start X
Sometimes the QEMU command does not exit after shutting down the VM.
### Windows version
Windows 10 Pro 1709 works for me, but 1803 does not (may be the UEFI). I have heard that the 1803 version comes with a Spectre patch and the performance is pathetic.
Windows 10 Pro 1709 works for me, but 1803 does not (may be the UEFI). [I have heard that the 1803 version comes with a Spectre patch and the performance is bad.](https://www.reddit.com/r/VFIO/comments/97unx4/passmark_lousy_2d_graphics_performance_on_windows/). The Spectre patch can be disabled.
## TODO
- [x] Unbind GPU without `virsh`
- [x] Update macos script
- [x] Unbind GPU without `virsh`.
- [x] Update macos script.
- [x] Run QEMU as user.
- [x] Try if is necessary to edit `/etc/mkinitcpio.conf`. No need to load the kernel modules at boot.
- [ ] Network
- [ ] Audio
- [ ] IOMMU guide
- [ ] Troubleshooting guide
- [ ] Extract the vBIOS in Linux guide
- [ ] How to edit the `windows.sh` script
- [ ] Fix the race condition
- [ ] Create scripts for install and use (Without DVD images)
- [x] Extract the vBIOS in Linux.
- [ ] Network.
- [ ] Install guide.
- [ ] IOMMU guide.
- [ ] Audio.
- [ ] Troubleshooting guide.
- [ ] How to edit the `windows.sh` script.
- [ ] Fix the race condition.
- [ ] Create scripts for install and use (Without DVD images).
- [ ] Improve the script with multiple options for HDD/IMG; network; PCI devices, etc.
- [ ] ACS Patch (Does not work for me).
- [ ] Install guide
- [ ] ???
- [ ] CPU pinning and RAM HugePages.
- [ ] Not kill X, [`shifter` & `xpra` may be the solution](https://www.linuxquestions.org/questions/linux-desktop-74/move-application-between-desktops-736982/#post4161705). Also [uswsusp (userspace software suspend)](https://wiki.archlinux.org/index.php/Uswsusp). [Source](https://www.reddit.com/r/linux_gaming/comments/98376e/i_am_creating_a_guide_for_gpu_passthrough_with/e4ebaoj/)
<!--
And you must supply QEMU with the Full GPU's ROM extracted extracted using a tool called "nvagetbios" , which you can find in a package called "envytools"
Yuri Alek
6 years ago