PRIME+VFIO on Sway with AMD+NVIDIA
PRIME offloading requires the GPU to be accessible to the host, so a few steps for setting up VFIO need to be undone. This is written specifically for my friend with this setup and may not be 100% transferrable if you're not running the same setup, so be careful if you've stumbled upon this page.
Remove IDs from bootloader config
This is undoing Step 3.1 from the arch wiki.
Edit your bootloader config (something like /boot/loader/entries/linux-lts-pt.conf
) and remove the VID:PIDs for the GPU in the kernel options for vfio-pci.ids
or pci-stub.ids
. You can find these easily with lspci -nnk | ack --passthru -i vga
.
While you're here, add the options:
-
initcall_blacklist=simpledrm_platform_driver_init
. This prevents a driver from being loaded that will randomize the order of /dev/dri/* devices that we need for a later step. -
nvidia_drm.modeset=1
andnvidia_drm.fbdev=1
. These do something.
Remove VFIO from mkinitcpio
This is undoing Step 3.2 from the arch wiki.
1) Delete /etc/modprobe.d/vfio.conf
if it exists.
2) Edit /etc/mkinitcpio.conf
and remove vfio_pci vfio vfio_iommu_type1
from the MODULES
section.
3) Rebuild your initramfs with sudo mkinitcpio -P
.
Prevent wlroots from using the GPU
Here is where we use the /dev/dri/* devices. By default, wlroots will assume you want to use every GPU for sway, which will use the outputs connected to the second GPU for sway and prevent the card from unbinding from the nvidia driver. The fix is pretty easy and can be set with an environment variable before sway starts. Edit your .zprofile (~/.config/zsh/.zprofile
) and put this line at the top of the file:
export WLR_DRM_DEVICES="/dev/dri/card0"
This tells wlroots to only use card0, which should be the card in the higher slot.
While you're here, you should also append --unsupported-gpu
to your exec sway
line so it reads exec sway --unsupported-gpu
. Sway won't start with the nvidia driver loaded by default, and this circumvents that.
Qemu hooks
The qemu hooks file lets you do stuff before a VM starts and when it shuts down.
Prevent the VM from starting if the GPU is in use
Add this to the prepare
section:
if [ -f "/dev/dri/card1" ]; then
if fuser -s "/dev/dri/card1" || fuser -s "/dev/dri/renderD129"; then
echo "Card is in use."
exit 1
fi
fi
Prevent wlroots from reclaiming the card when the VM is shut down
Add this to the release
section:
udevadm trigger --verbose --type=devices --action=remove --subsystem-match=drm --property-match="MINOR=1"
Install necessary packages
Install the nvidia driver and other necessary packages with sudo pacman -S nvidia nvidia-lts nvidia-utils nvidia-prime
.
Reboot
Rebooting will load all of these changes. The nvidia driver should be loaded at boot but sway should only be using the AMD card.
Test with prime-run
Any program can be started with prime-run to use the nvidia card. You can test this with:
prime-run glxinfo | grep "OpenGL renderer"