Archives
/
vfio-guide-2
mirror of https://gitlab.com/YuriAlek/vfio
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Less sleep

Browse Source
testing-auto
yurialek 6 years ago
parent d6a1f7d1c6
commit 49f23ddea7
6 changed files with 33 additions and 94 deletions
Show Stats Download Patch File Download Diff File

2
Hardware configurations/README.md
Unescape Escape View File

@ -38,7 +38,7 @@ Ryzen 5 2600 - GTX 770 - @YuriAlek.md
### How I did it ### How I did it
For extracting the `vBIOS` I used the 1st method in Linux and edited it. For extracting the `vBIOS` I used the 1st method in Linux and edited it.
You can add anything that you consider useful like the step needed in your distro for installing everything. The more information, and solutions, the better. You can add anything that you consider useful like the steps needed in your distro for installing everything or a link to your script. The more information, and solutions, the better.
### Files to modify ### Files to modify
#### `scripts/windows.sh` #### `scripts/windows.sh`

12
Install.md
Unescape Escape View File

@ -6,7 +6,7 @@
``` ```
### Enabling IOMMU support on boot ### Enabling IOMMU support on boot
For `systemd-boot` edit `/boot/loader/entries/arch.conf` and add `intel_iommu=on` **OR** `amd_iommu=on` and `iommu=pt`. #### For `systemd-boot` edit `/boot/loader/entries/arch.conf` and add `intel_iommu=on` **OR** `amd_iommu=on` and `iommu=pt`.
``` ```
# AMD # AMD
options root=/dev/sda2 amd_iommu=on iommu=pt options root=/dev/sda2 amd_iommu=on iommu=pt
@ -15,8 +15,8 @@ options root=/dev/sda2 amd_iommu=on iommu=pt
options root=/dev/sda2 intel_iommu=on iommu=pt options root=/dev/sda2 intel_iommu=on iommu=pt
``` ```
Reboot. Reboot.
----
For `GRUB` edit `/etc/default/grub` and append your kernel options to the `GRUB_CMDLINE_LINUX_DEFAULT`. #### For `GRUB` edit `/etc/default/grub` and append your kernel options, `intel_iommu=on` **OR** `amd_iommu=on` and `iommu=pt`, to the `GRUB_CMDLINE_LINUX_DEFAULT`.
``` ```
# AMD # AMD
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt" GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt"
@ -29,10 +29,10 @@ And then automatically re-generate the grub.cfg file with:
# grub-mkconfig -o /boot/grub/grub.cfg # grub-mkconfig -o /boot/grub/grub.cfg
``` ```
Reboot. Reboot.
----
After reboot IOMMU should be working After reboot IOMMU should be working:
``` ```
[yu@ryzen ~]$ dmesg | grep -e DMAR -e IOMMU $ dmesg | grep -e DMAR -e IOMMU
-------------------------------------------- --------------------------------------------
[ 0.492684] AMD-Vi: IOMMU performance counters supported [ 0.492684] AMD-Vi: IOMMU performance counters supported
[ 0.494370] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 0.494370] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

23
README.md
Unescape Escape View File

@ -13,20 +13,23 @@
7. [Known problems](#known-problems) 7. [Known problems](#known-problems)
8. [TODO](#todo) 8. [TODO](#todo)
## Last Update
2018/10/20
## What this does ## What this does
In one command it kills X, frees the GPU from drivers and console, detaches the GPU from the host, starts the VM with the GPU, waits until the VM is off, reattaches the GPU to the host and starts lightdm. In one command it kills X, frees the GPU from drivers and console, detaches the GPU from the host, starts the VM with the GPU, waits until the VM is off, reattaches the GPU to the host and starts lightdm.
## What you need ## What you need
* An IOMMU enabled motherboard. Check your motherboard manual for an option in BIOS to enable IOMMU. * An IOMMU enabled motherboard. Check your motherboard manual for an option in BIOS to enable IOMMU.
* CPU support for AMD-v/VT-x and AMD-Vi/VT-d (AMD/Intel). * CPU support for AMD-v/VT-x and AMD-Vi/VT-d (AMD/Intel).
* Virtualization support enabled on BIOS; check your motherboard manual if you need help. * Virtualization support enabled on BIOS. Check your motherboard manual if you need help.
* One GPU that supports UEFI and its BIOS. All GPUs from 2012 and later should support this, some may have issues. If the GPU does not support UEFI you may be able to make it work, but you won't see anything in the screen until the drivers inside Windows kick in. * One GPU that supports UEFI and its BIOS. All GPUs from 2012 and later should support this, some may have issues. If the GPU does not support UEFI you may be able to make it work, but you won't see anything in the screen until the drivers inside Windows kick in.
* QEMU, OVMF UEFI and VIRTIO drivers for Windows. [If you need to install, refer to the Install guide](Install.md) * QEMU, OVMF UEFI and VIRTIO drivers for Windows. [If you need to install, refer to the Install guide](Install.md)
* [Optional] HDD only for Windows * [Optional] An hard drive only for Windows.
* [Recommended] Another computer to login remotely with `ssh` for convenience, at least until you have everything working. * [Recommended] Another computer to login remotely with `ssh` for convenience, at least until you have everything working.
## Submit your own configuration ## Submit your own configuration
[Create a pull merge](https://gitlab.com/YuriAlek/vfio/merge_requests/new) with [a file explaining how you got it working](Hardware configurations/README.md) [Create a pull merge](https://gitlab.com/YuriAlek/vfio/merge_requests/new) with [a file explaining how you got it working](Hardware configurations/README.md).
## My system ## My system
``` ```
@ -42,14 +45,14 @@ In one command it kills X, frees the GPU from drivers and console, detaches the
[Software] [Software]
Linux Distro: ArchLinux Linux Distro: ArchLinux
Linux Kernel: 4.17.14 vanilla Linux Kernel: 4.17.14
Nvidia divers: 396.51-1 Nvidia divers: 396.51-1
QEMU version: 2.12.1-1 QEMU version: 2.12.1-1
OVMF version: r24021 OVMF version: r24021
[Guests] [Guests]
Windows: Windows 10 Pro 1709 x64 Windows 10 Pro 1709 x64
MacOS: MacOS High Sierra 10.13.3 MacOS High Sierra 10.13.3
``` ```
## vBIOS ## vBIOS
@ -119,6 +122,7 @@ $ qemu-img create -f raw /path/to/image/windows.raw 60G
``` ```
6. Edit the config in `scripts/config.sh` to convenience. If you use systemd to start the VM you have to edit `EnvironmentFile` in `qemu@.service` to point to your config file. Variables you may have to edit: 6. Edit the config in `scripts/config.sh` to convenience. If you use systemd to start the VM you have to edit `EnvironmentFile` in `qemu@.service` to point to your config file. Variables you may have to edit:
1. PCI devices. `IOMMU_GPU`; `IOMMU_USB`. 1. PCI devices. `IOMMU_GPU`; `IOMMU_USB`.
2. User. 2. User.
3. Location of HDD/IMG, ISO, vBIOS and OVMF image. 3. Location of HDD/IMG, ISO, vBIOS and OVMF image.
@ -135,6 +139,7 @@ $ qemu-img create -f raw /path/to/image/windows.raw 60G
``` ```
8. When installing Windows, in the section `Where do you want to install Windows?` there will be no hard drives to install to; to fix it: 8. When installing Windows, in the section `Where do you want to install Windows?` there will be no hard drives to install to; to fix it:
1. Load driver 1. Load driver
2. Browse 2. Browse
3. CD Drive (E:) virtio-win-0.1.1 3. CD Drive (E:) virtio-win-0.1.1
@ -192,17 +197,17 @@ Windows 10 Pro 1709 works for me, but 1803 does not (may be the UEFI). [I have h
## TODO ## TODO
- [x] Unbind GPU without `virsh`. - [x] Unbind GPU without `virsh`.
- [x] Update macos script.
- [x] Run QEMU as user. - [x] Run QEMU as user.
- [x] Try if is necessary to edit `/etc/mkinitcpio.conf`. No need to load the kernel modules at boot. - [x] Try if is necessary to edit `/etc/mkinitcpio.conf`. No need to load the kernel modules at boot.
- [x] Extract the vBIOS in Linux. - [x] Extract the vBIOS in Linux.
- [x] Install guide. - [x] Install guide.
- [ ] Network. - [x] Fix the race condition.
- [ ] Network. [I am too lazy, check this](https://yurialek.gitlab.io/gitbook/docs/qemu/network.html). I will add it to the scripts at some point.
- [ ] Update macos script.
- [ ] IOMMU guide. - [ ] IOMMU guide.
- [ ] Audio. - [ ] Audio.
- [ ] Troubleshooting guide. - [ ] Troubleshooting guide.
- [ ] How to edit the `windows.sh` script. - [ ] How to edit the `windows.sh` script.
- [ ] Fix the race condition.
- [ ] Create scripts for install and use (Without DVD images). - [ ] Create scripts for install and use (Without DVD images).
- [ ] Improve the script with multiple options for HDD/IMG; network; PCI devices, etc. - [ ] Improve the script with multiple options for HDD/IMG; network; PCI devices, etc.
- [ ] ACS Patch (Does not work for me). - [ ] ACS Patch (Does not work for me).

0
scripts/iommu.sh
Unescape Escape View File

43
scripts/windows-install.sh
Unescape Escape View File

@ -10,8 +10,8 @@ fi
source config source config
# Memory lock limit # Memory lock limit
if [ $(ulimit -a | grep "max locked memory" | awk '{print $6}') != 12884900 ]; then if [ $(ulimit -a | grep "max locked memory" | awk '{print $6}') != $(( $(echo $RAM | tr -d 'G')*1048576+10 )) ]; then
ulimit -l 12884900 ulimit -l $(( $(echo $RAM | tr -d 'G')*1048576+10 ))
fi fi
## Kill X and related ## Kill X and related
@ -21,58 +21,35 @@ sleep 2
# Kill the console to free the GPU # Kill the console to free the GPU
echo 0 > /sys/class/vtconsole/vtcon0/bind echo 0 > /sys/class/vtconsole/vtcon0/bind
sleep 1
echo 0 > /sys/class/vtconsole/vtcon1/bind echo 0 > /sys/class/vtconsole/vtcon1/bind
sleep 1
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
sleep 1
# Unload the Kernel Modules that use the GPU # Unload the Kernel Modules that use the GPU
modprobe -r nvidia_drm modprobe -r nvidia_drm
sleep 1
modprobe -r nvidia_modeset modprobe -r nvidia_modeset
sleep 1
modprobe -r nvidia modprobe -r nvidia
sleep 1
modprobe -r snd_hda_intel modprobe -r snd_hda_intel
sleep 2
# Load the kernel module # Load the kernel module
modprobe vfio modprobe vfio
sleep 1
modprobe vfio_iommu_type1 modprobe vfio_iommu_type1
sleep 1
modprobe vfio-pci modprobe vfio-pci
sleep 1
# Detach the GPU from drivers and attach to vfio. Also the usb. # Detach the GPU from drivers and attach to vfio. Also the usb.
echo $videoid > /sys/bus/pci/drivers/vfio-pci/new_id echo $videoid > /sys/bus/pci/drivers/vfio-pci/new_id
sleep 1
echo $videobusid > /sys/bus/pci/devices/$videobusid/driver/unbind echo $videobusid > /sys/bus/pci/devices/$videobusid/driver/unbind
sleep 1
echo $videobusid > /sys/bus/pci/drivers/vfio-pci/bind echo $videobusid > /sys/bus/pci/drivers/vfio-pci/bind
sleep 1
echo $videoid > /sys/bus/pci/drivers/vfio-pci/remove_id echo $videoid > /sys/bus/pci/drivers/vfio-pci/remove_id
sleep 1
echo $audioid > /sys/bus/pci/drivers/vfio-pci/new_id echo $audioid > /sys/bus/pci/drivers/vfio-pci/new_id
sleep 1
echo $audiobusid > /sys/bus/pci/devices/$audiobusid/driver/unbind echo $audiobusid > /sys/bus/pci/devices/$audiobusid/driver/unbind
sleep 1
echo $audiobusid > /sys/bus/pci/drivers/vfio-pci/bind echo $audiobusid > /sys/bus/pci/drivers/vfio-pci/bind
sleep 1
echo $audioid > /sys/bus/pci/drivers/vfio-pci/remove_id echo $audioid > /sys/bus/pci/drivers/vfio-pci/remove_id
sleep 1
echo $usbid > /sys/bus/pci/drivers/vfio-pci/new_id echo $usbid > /sys/bus/pci/drivers/vfio-pci/new_id
sleep 1
echo $usbbusid > /sys/bus/pci/devices/$usbbusid/driver/unbind echo $usbbusid > /sys/bus/pci/devices/$usbbusid/driver/unbind
sleep 1
echo $usbbusid > /sys/bus/pci/drivers/vfio-pci/bind echo $usbbusid > /sys/bus/pci/drivers/vfio-pci/bind
sleep 1
echo $usbid > /sys/bus/pci/drivers/vfio-pci/remove_id echo $usbid > /sys/bus/pci/drivers/vfio-pci/remove_id
#ls -la /sys/bus/pci/devices/$usbbusid/
sleep 1
# QEMU (VM) command # QEMU (VM) command
qemu-system-x86_64 -runas $USER -enable-kvm \ qemu-system-x86_64 -runas $USER -enable-kvm \
@ -99,31 +76,23 @@ qemu-system-x86_64 -runas $USER -enable-kvm \
# Wait for QEMU to finish before continue # Wait for QEMU to finish before continue
wait wait
sleep 5 sleep 1
# Unload the vfio module. I am lazy, this leaves the GPU without drivers # Unload the vfio module. I am lazy, this leaves the GPU without drivers
modprobe -r vfio-pci modprobe -r vfio-pci
sleep 2
modprobe -r vfio_iommu_type1 modprobe -r vfio_iommu_type1
sleep 2
modprobe -r vfio modprobe -r vfio
sleep 2
# Reload the kernel modules. This loads the drivers for the GPU # Reload the kernel modules. This loads the drivers for the GPU
modprobe snd_hda_intel modprobe snd_hda_intel
sleep 5
modprobe nvidia_drm modprobe nvidia_drm
sleep 2
modprobe nvidia_modeset modprobe nvidia_modeset
sleep 2
modprobe nvidia modprobe nvidia
sleep 5
# Bind the usb # Bind the usb
#echo $usbid > /sys/bus/pci/drivers/xhci_hcd/new_id #echo $usbid > /sys/bus/pci/drivers/xhci_hcd/new_id
echo $usbbusid > /sys/bus/pci/devices/$usbbusid/driver/unbind echo $usbbusid > /sys/bus/pci/devices/$usbbusid/driver/unbind
echo $usbbusid > /sys/bus/pci/drivers/xhci_hcd/bind echo $usbbusid > /sys/bus/pci/drivers/xhci_hcd/bind
sleep 10
#echo $usbid > /sys/bus/pci/drivers/xhci_hcd/remove_id #echo $usbid > /sys/bus/pci/drivers/xhci_hcd/remove_id
#ls -la /sys/bus/pci/devices/$usbbusid/ #ls -la /sys/bus/pci/devices/$usbbusid/
@ -132,13 +101,11 @@ sleep 10
echo 1 > /sys/class/vtconsole/vtcon0/bind echo 1 > /sys/class/vtconsole/vtcon0/bind
sleep 1 sleep 1
echo 1 > tee /sys/class/vtconsole/vtcon1/bind echo 1 > tee /sys/class/vtconsole/vtcon1/bind
sleep 5 sleep 1
#echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
#sleep 1
# Reload the Display Manager to access X # Reload the Display Manager to access X
systemctl start lightdm systemctl start lightdm
sleep 5 sleep 2
# Restore the Frame Buffer # Restore the Frame Buffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind

47
scripts/windows.sh
Unescape Escape View File

@ -7,11 +7,11 @@ if [ "$EUID" -ne 0 ]
fi fi
# END Check if you are sudo # END Check if you are sudo
source /home/yu/scripts/config source config
# Memory lock limit # Memory lock limit.
if [ $(ulimit -a | grep "max locked memory" | awk '{print $6}') != 12884900 ]; then if [ $(ulimit -a | grep "max locked memory" | awk '{print $6}') != $(( $(echo $RAM | tr -d 'G')*1048576+10 )) ]; then
ulimit -l 12884900 ulimit -l $(( $(echo $RAM | tr -d 'G')*1048576+10 ))
fi fi
## Kill X and related ## Kill X and related
@ -21,58 +21,35 @@ sleep 2
# Kill the console to free the GPU # Kill the console to free the GPU
echo 0 > /sys/class/vtconsole/vtcon0/bind echo 0 > /sys/class/vtconsole/vtcon0/bind
sleep 1
echo 0 > /sys/class/vtconsole/vtcon1/bind echo 0 > /sys/class/vtconsole/vtcon1/bind
sleep 1
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
sleep 1
# Unload the Kernel Modules that use the GPU # Unload the Kernel Modules that use the GPU
modprobe -r nvidia_drm modprobe -r nvidia_drm
sleep 1
modprobe -r nvidia_modeset modprobe -r nvidia_modeset
sleep 1
modprobe -r nvidia modprobe -r nvidia
sleep 1
modprobe -r snd_hda_intel modprobe -r snd_hda_intel
sleep 2
# Load the kernel module # Load the kernel module
modprobe vfio modprobe vfio
sleep 1
modprobe vfio_iommu_type1 modprobe vfio_iommu_type1
sleep 1
modprobe vfio-pci modprobe vfio-pci
sleep 1
# Detach the GPU from drivers and attach to vfio. Also the usb. # Detach the GPU from drivers and attach to vfio. Also the usb.
echo $videoid > /sys/bus/pci/drivers/vfio-pci/new_id echo $videoid > /sys/bus/pci/drivers/vfio-pci/new_id
sleep 1
echo $videobusid > /sys/bus/pci/devices/$videobusid/driver/unbind echo $videobusid > /sys/bus/pci/devices/$videobusid/driver/unbind
sleep 1
echo $videobusid > /sys/bus/pci/drivers/vfio-pci/bind echo $videobusid > /sys/bus/pci/drivers/vfio-pci/bind
sleep 1
echo $videoid > /sys/bus/pci/drivers/vfio-pci/remove_id echo $videoid > /sys/bus/pci/drivers/vfio-pci/remove_id
sleep 1
echo $audioid > /sys/bus/pci/drivers/vfio-pci/new_id echo $audioid > /sys/bus/pci/drivers/vfio-pci/new_id
sleep 1
echo $audiobusid > /sys/bus/pci/devices/$audiobusid/driver/unbind echo $audiobusid > /sys/bus/pci/devices/$audiobusid/driver/unbind
sleep 1
echo $audiobusid > /sys/bus/pci/drivers/vfio-pci/bind echo $audiobusid > /sys/bus/pci/drivers/vfio-pci/bind
sleep 1
echo $audioid > /sys/bus/pci/drivers/vfio-pci/remove_id echo $audioid > /sys/bus/pci/drivers/vfio-pci/remove_id
sleep 1
echo $usbid > /sys/bus/pci/drivers/vfio-pci/new_id echo $usbid > /sys/bus/pci/drivers/vfio-pci/new_id
sleep 1
echo $usbbusid > /sys/bus/pci/devices/$usbbusid/driver/unbind echo $usbbusid > /sys/bus/pci/devices/$usbbusid/driver/unbind
sleep 1
echo $usbbusid > /sys/bus/pci/drivers/vfio-pci/bind echo $usbbusid > /sys/bus/pci/drivers/vfio-pci/bind
sleep 1
echo $usbid > /sys/bus/pci/drivers/vfio-pci/remove_id echo $usbid > /sys/bus/pci/drivers/vfio-pci/remove_id
#ls -la /sys/bus/pci/devices/$usbbusid/
sleep 1
# QEMU (VM) command # QEMU (VM) command
qemu-system-x86_64 -runas $USER -enable-kvm \ qemu-system-x86_64 -runas $USER -enable-kvm \
@ -95,31 +72,23 @@ qemu-system-x86_64 -runas $USER -enable-kvm \
# Wait for QEMU to finish before continue # Wait for QEMU to finish before continue
wait wait
sleep 5 sleep 1
# Unload the vfio module. I am lazy, this leaves the GPU without drivers # Unload the vfio module. I am lazy, this leaves the GPU without drivers
modprobe -r vfio-pci modprobe -r vfio-pci
sleep 2
modprobe -r vfio_iommu_type1 modprobe -r vfio_iommu_type1
sleep 2
modprobe -r vfio modprobe -r vfio
sleep 2
# Reload the kernel modules. This loads the drivers for the GPU # Reload the kernel modules. This loads the drivers for the GPU
modprobe snd_hda_intel modprobe snd_hda_intel
sleep 5
modprobe nvidia_drm modprobe nvidia_drm
sleep 2
modprobe nvidia_modeset modprobe nvidia_modeset
sleep 2
modprobe nvidia modprobe nvidia
sleep 5
# Bind the usb # Bind the usb
#echo $usbid > /sys/bus/pci/drivers/xhci_hcd/new_id #echo $usbid > /sys/bus/pci/drivers/xhci_hcd/new_id
echo $usbbusid > /sys/bus/pci/devices/$usbbusid/driver/unbind echo $usbbusid > /sys/bus/pci/devices/$usbbusid/driver/unbind
echo $usbbusid > /sys/bus/pci/drivers/xhci_hcd/bind echo $usbbusid > /sys/bus/pci/drivers/xhci_hcd/bind
sleep 10
#echo $usbid > /sys/bus/pci/drivers/xhci_hcd/remove_id #echo $usbid > /sys/bus/pci/drivers/xhci_hcd/remove_id
#ls -la /sys/bus/pci/devices/$usbbusid/ #ls -la /sys/bus/pci/devices/$usbbusid/
@ -128,13 +97,11 @@ sleep 10
echo 1 > /sys/class/vtconsole/vtcon0/bind echo 1 > /sys/class/vtconsole/vtcon0/bind
sleep 1 sleep 1
echo 1 > tee /sys/class/vtconsole/vtcon1/bind echo 1 > tee /sys/class/vtconsole/vtcon1/bind
sleep 5 sleep 1
#echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
#sleep 1
# Reload the Display Manager to access X # Reload the Display Manager to access X
systemctl start lightdm systemctl start lightdm
sleep 5 sleep 2
# Restore the Frame Buffer # Restore the Frame Buffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind

Write Preview
Loading…
Cancel
Save