QEMU GPU vfio-pci pass through

@Balaton
that's right,
I'm missing the last step

the steps:

I've learned it by heart now, I've tried at least ten distributions, I've also used Liquorix as a kernel and also XanMod to try

lspci -nn
Model Radeon GPU etc. etc.

/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio-pci.ids=1002:6798,1002:aaa0 quiet splash" <----- there is also a third parameter to add that I can't find anymore but I have it somewhere that greatly increases the speed of the GPU in emulation with vfio.
I read it in a linux forum

sudo grub-mkconfig -o /boot/grub/grub.cfg

reboot

/etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:6798,1002:aaa0
softdep Radeon pre: vfio-pci

sudo update-initramfs -c -k $(uname -r)

reboot

I'm missing the last step to put the Radeon GPU in Black-List
I don't understand in Ubuntu in which file I have to write to put the Radeon GPU in Black-List

note:
I guess it doesn't matter if it is installed after qemu or is it better to install first qemu ?

Thanks

I read bboot.fth I found it in the link you gave me.

Now the question is why have you tried so many distributions?
Because Ubuntu 24.10 is practically full of bugs.
In all its variants lubuntu xbuntu etc.

The one that has not given me problems is the beta
Ubuntu 25.04 "Plucky Puffin"
It works without problems recognizes everything does not give problems of any kind should be released on April 17.

I found this
Now let's see I should work better

If any user has managed to do it in another way it would be nice to know

I like to write what I do so as to see if I can be helped


https://www.heiko-sieger.info/blacklisting-graphics-driver/

@Balaton

I tried this script on the fly
sudo apt install driverctl

then
driverctl set-override 0000:03:00.0 vfio-pci
driverctl set-override 0000:03:00.1 vfio-pci

this is the result now it is associated I hope it is the right way:



you can take a look at this script and maybe adapt it for our purposes:
https://gitlab.com/driverctl/driverctl

I will return to work on it tonight

@white
Quote:
If nothing else this is a good opportunity to try things and learn. That's worth even when the result may not be ready yet but you can do something new on the way.

Quote:
You don't need vfio-pci.ids here unless you have vfio-pci built in the kernel instead of being a module but I don't think any distro does that. The only option really needed here is amd_iommu=on (or intel_iommu=on if you have Intel CPU). The iommu=pt may or may not help, it's less secure than without it but may be faster. Other vfio options can go in /etc/modprobe.d

Quote:
I think you need sotfdep drm and softdep snd_hda_intel instead of Radeon here as I wrote before. Those lines tell modprobe to load vfio-pci before the drm and snd_hda_intel drivers so it can bind to the GPU instead of those drivers, otherwise it would be loaded too late when the GPU driver already took the card. If your GPU uses amdgpu you may need softdep amdgpu pre: vfio-pci instead but drm should take care of those as it's loaded before any GPU drivers. You can also add "blacklist amdgpu" to this file but only if your secondary and primary GPUs are different otherwise you would lose graphics on your host too (in your case it works as you have NVidia for the host and AMD for guest).

I don't know what driverctl does but if that achieved the same to have the card to pass through driven by vfio-pci then that's OK but the above would make it permanent and you don't have to change that on every reboot.

@Balaton

Using the driverctl utility

driverctl is a relatively new Linux device driver control utility that was developed for recent Fedora or RHEL 7 distributions, but is also available under Linux Mint 20 (and most likely all new Ubuntu 20.04 based distros). This utility integrates with udev and makes driver assignment a breeze.

To install, use:

sudo apt install driverctl

A description of this utility (as well as the sources) can be found here:

https://gitlab.com/driverctl/driverctl

Using the PCI bus numbers we determined above, enter the following commands in a root shell to bind the graphics card to the vfio-pci driver:

driverctl set-override 0000:02:00.0 vfio-pci
driverctl set-override 0000:02:00.1 vfio-pci

To unbind, use the following command:

driverctl unset-override 0000:02:00.0
driverctl unset-override 0000:02:00.1

Please refer to the link above and the examples given in the README for more information.

Maybe you missed the link I posted:

https://www.heiko-sieger.info/blacklisting-graphics-driver/

@white
Quote:Wont work unless you blacklist the amdgpu (or other Linux ATI) driver.
It a Linux AMD/ATI driver is started it initializes the ATI gfx card which usually results is a state the AmigaOS 4.x Radeon HD/RX drivers can't work with.

@white

I received a card from a friend today.



This card definitely works on AOS4. It's not RTX, but it might be easier to use.
Best of all, to test this card I had to dig out my Hackintosh ... I booted up Mac OS X 10.7 Lion and the shock... How this system runs beautifully, fast on HDD
Sometimes you go back 11 years and things don't look the same. But in this case, the opposite is true.
The tested ASUS HD 7750 V2 card works. I have the stupid idea of trying to run AOS4 with this card on a PC from the time the X1000 came out.
It's 99.9% impossible on that old motherboard. There is always that 0.1% which is a nice challenge and a nice time

Regards!

@joerg
I'm looking for the easiest way and even if I get to AmigaOS then there will be other things to do.
Anyway I will try the various ways.
I hope to receive support.

@smarkusg
The first time I emulated hackintosh I did it on an Asus NetBook
one of the first versions worked,
but I had no software and I didn't want to start downloading programs that I wouldn't have used anyway.
They are always fun things even just to try them.

NOTE:
please check if my command line is correct, thanks


Something has changed now I still haven't blacklisted with the editor manually.

Now he says this:
sudo qemu-system-ppc -M pegasos2 -kernel /home/white/Downloads/bboot-0.8/bboot -initrd /home/white/Downloads/Kickstart.zip -vga none -device VGA,romfile="" -m 2048 -drive if=none,id=hd,file=/home/white/Downloads/32gb.raw,format=raw -device ide-hd,drive=hd,bus=ide.1 -drive if=none,id=hd1,file=/home/white/Scaricati/coffin.raw,format=raw -device ide-hd,drive=hd1,bus=ide.1 -device rtl8139,netdev=network01 -netdev user,id=network01 -device vfio-pci,host=03:00.0,bus=pci.0,x-vga=on -device vfio-pci.host=03:00.1,bus=pci.0 -accel tcg

qemu-system-ppc: -device vfio-pci,host=03:00.0,bus=pci.0,x-vga=on: vfio 0000:03:00.0: group 14 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

Also I probably need to add the AUDIO in "vfio.conf"
because I saw it says something about INTEL

what should I add for the intel audio?

here are the commands:
lspci -k|grep vfio
Kernel driver in use: vfio-pci
Kernel driver in use: vfio-pci

lspci -vvs 03:00.0
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] (prog-if 00 [VGA controller])
Subsystem: PC Partner Limited / Sapphire Technology Device 3001
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 32
IOMMU group: 14
Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256M]
Region 2: Memory at fc600000 (64-bit, non-prefetchable) [size=256K]
Region 4: I/O ports at e000 [size=256]
Expansion ROM at fc640000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: radeon, amdgpu

lspci -vvs 03:00.1
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970]
Subsystem: PC Partner Limited / Sapphire Technology Device aaa0
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 35
IOMMU group: 14
Region 0: Memory at fc660000 (64-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel <<<------THIS

lspci -nnks 03:00
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
Subsystem: PC Partner Limited / Sapphire Technology Device [174b:3001]
Kernel driver in use: vfio-pci
Kernel modules: radeon, amdgpu
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]
Subsystem: PC Partner Limited / Sapphire Technology Device [174b:aaa0]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel <<<---- THIS

Kernel modules: snd_hda_intel <<<-----THIS

what should I add in vfio.conf ?

The audio device at 03:00.1 is probably in the same IOMMU group and needs to be passed through to the guest along with the GPU. That audio device would be the HDMI audio out of the graphics card.

@Balaton
you already wrote it to me:

softdep snd_hda_intel pre: vfio-pci

instead this is my vfio.conf

options vfio-pci ids=1002:6798,1002:aaa0
softdep Radeon pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci

now I try..

here is a report with:

sudo dmesg |grep AMD-Vi

[ 0.036224] AMD-Vi: Unknown option - 'on'
[ 0.103384] AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0
[ 0.705557] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.706873] AMD-Vi: Completion-Wait loop timed out
[ 0.833191] AMD-Vi: Completion-Wait loop timed out
[ 0.958374] AMD-Vi: Completion-Wait loop timed out
[ 1.083511] AMD-Vi: Completion-Wait loop timed out
[ 1.208616] AMD-Vi: Completion-Wait loop timed out
[ 1.333705] AMD-Vi: Completion-Wait loop timed out
[ 1.458803] AMD-Vi: Completion-Wait loop timed out
[ 1.706678] AMD-Vi: Extended features (0x58f77ef22294a5a, 0x0): PPR NX GT IA PC GA_vAPIC
[ 1.706686] AMD-Vi: Interrupt remapping enabled
[ 1.706778] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002e8900]

@white
It's hard to tell what's your problem with only seeing random bits of information but from the lspci outputs:
Quote:
I think you need something like this:

The last two blacklist lines may not actually be needed but would make sure these drivers are not even loaded. Since your other card is NVidia these should not be needed so you can blacklist them.

The HDA standard was made by Intel so the driver is called like that even if your implementation is from AMD, you still need to make sure that the audio part of the card is also bound to vfio. If you still get the same error then check what's in the same iommu group. There are scripts for that but a simple one is:

If you get an error it helps to search for that error and you can usually find solutions. Here is one:
https://vfio.blogspot.com/2014/08/vfiovga-faq.html
but this is old and the patches it mentions is likely not recommended, just make sure you assign all devices in the same iommu group to vfio-pci. You may need to put the card in a different slot to get an iommu group with only the card in it. Read the link under "previous post about IOMMU groups" in the above article.

As for your command line you can't use -kernel bboot with vfio-pci. You'll need -bios pegasos2.rom and load bboot from the guest's disk from within the pegasos firmware. (It may appear to work with just -kernel bboot because the AmigaOS driver tries to run the AtomBIOS but without the card's ROM running first that may not fully init the card so you need something that runs the card ROM and BBoot does not do that yet. Eventually I may add that to bboot too but for now you need the machine firmware to run the card ROM then need bboot.fth on pegasos2 to patch bugs of the pegasos2 AmigaOS kernel for which no update is available yet.)

@joerg
Quote:
It should work even if driverctl just unbinds and not modifies the config to add the pre: clauses because the card is reset when QEMU starts unless the card cannot be reset for some reason or has issues like mentioned here: https://askubuntu.com/questions/121166 ... ure-all-devices-within-th but that's for NVidia not ATI. But it is more likely to work if the card is as it powered up and not touched by anything before the machine firmware has a chance to run its ROM so going for that has the highest chance to work.

@balaton
Quote:I don't know anything about vfio-pci and iommu on Linux, but the following may be a problem 0:3:0.0 and 0:3:0.1 is the ATI gfx card.
I guess it needs to be in a separate iommu group, with nothing else in it (0:1:x.y, 0:2:x.y and 0:5:x.y), to work?