QEMU GPU vfio-pci pass through

@tlosm
Quote:May be a problem on a real PegasosII, but @balaton fixed all problems (by converting 64 bit BARs to 32 bit, interrupt related fixes, etc.) in his bboot.fth and BBoot, and probably in QEmu itself as well.
There should be no problems using the QEmu emulation of the PegasosII any more with PCIe gfx cards, at least not with AmigaOS 4.x, MorphOS may be different.
Using BBoot on a real PegasosII is probably not possible, but using bboot.fth should be, and it might even help getting PCIe gfx cards working with a PCI<->PCIe bridge.
The real AmigaOne SE/XE/µA1 systems have at least 100 times as much problems, hardware limits and bugs as the QEmu emulation of it, but QEmu doesn't emulate the hardware limits and bugs of the PegasosII and AmigaOne SE/XE/µA1

Several people got Radeon HD/RX PCIe cards working with vfio-pci with QEmu, and all used the PegasosII emulation since it used to be the best one for AmigaOS 4.x.
Now, with NVRAM support etc. added, the AmigaOne XE emulation is probably better than the PegasosII one for AmigaOS 4.x.
Sam4x0 emulation is too slow, one of the reasons is the missing "real" MMU, using TLB caches instead, which cause a lot of exceptions, and which probably can't be fixed/worked around in the QEmu emulation.
However, only @geenaam got vfio-pci working with usable speeds, for everyone else it was extremely slow, the real Radeon PCIe gfx card was even much slower than using SM501/502 software emulation.

What @white has to fix are the iommu groups, the Radeon gfx card has to be in a separate group with nothing else (required by the host) in it.
The easiest, and maybe only, option is trying to use a different PCIe slot for the Radeon gfx card.

@joerg
Quote:

Just to clarify a bit: I use bboot on real pegasos2 and it works fine. It speed up loading of os4 for about 10 seconds (30s vs 40s with os4s loader)

And, while maybe not very important at the moment because of "beta":

Quote:

In beta kernel for os4 all this also fixed already, and i can use real RadeonRx and RadeonHd via bridges, but.. that not released for now, sadly.

@joerg

Quote:

Swapping PCI slots is one way. The other is to use ACS-overwrite. This requires both a new kernel and some lines in the grub boot. I don't have the link handy that tells how to do that, butas for the ACS overwrite/patch, you just need ot install linux-zen as the kernel (I think I did that on arch linux using pacman -Sy linux-zen or similar... can be found online). Think it is similar on ubuntu-based systems.

After that, and reboot, etc, you will see only a single pci device per IOMMU group (you will have many groups). You can then just make a '-device ' entry in the qemu command line for just the video part and not bother with the audio if you don't want.

@themamboman

CachyOS based on Arch and its Linux kernels also have similar functionality.

https://wiki.cachyos.org/cachyos_basic/why_cachyos/
https://github.com/CachyOS/kernel-patches

The Linux kernel for the desktop in popular distributions is best recompiled yourself.
The versions included in the distributions are too (so to speak...) neutral.

Current situation
It seems that ACS with the Liquorix kernel is now working or so it seems.
I always write what I do so as to keep a reference page in the Forum.

Steps:
uname -r
6.14. 0-1-liquorix-amd64

then use the Kernel Stub:

sudo kernelstub --add-options "pcie_acs_override=downstream,multifunction" <<<----- THIS ENABLES ACS IOMMU with Liquorix kernel

sudo update-initramfs -u


reboot

It seems that ACS is now enabled with the Liquorix Kernel

for d in $(find /sys/kernel/iommu_groups/ -type l | sort -n -k5 -t/); do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;

New IOMMU Result:

IOMMU Group 0 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
IOMMU Group 1 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU Group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
IOMMU Group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
IOMMU Group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
IOMMU Group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
IOMMU Group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
IOMMU Group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
IOMMU Group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
IOMMU Group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
IOMMU Group 14 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
IOMMU Group 15 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
IOMMU Group 16 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
IOMMU Group 17 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 18 02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 19 02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 20 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
IOMMU Group 21 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]
IOMMU Group 22 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 04)
IOMMU Group 23 06:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2882] (rev a1)
IOMMU Group 24 06:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22be] (rev a1)
IOMMU Group 25 07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 26 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 27 08:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 28 08:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 29
08:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
white@pop-os:~$

Note:
I guess I need to reconfigure the ALIAS now

Now the Radeon card is here IOMMU group 20 and 21:

IOMMU Group 20 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
IOMMU Group 21 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]

--------------------------------------------------------------

If you want, you can also manually edit this file:
/etc/kernelstub

{
"default": {
"kernel_options": [
"quiet",
"splash"
],
"esp_path": "/boot/efi",
"setup_loader": false,
"manage_mode": false,
"force_update": false,
"live_mode": false,
"config_rev": 3
},
"user": {
"kernel_options": [
"vfio-pci.ids",
"pcie_acs_override=downstream,multifunction"
],
"esp_path": "/boot/efi",
"setup_loader": true,
"manage_mode": true,
"force_update": false,
"live_mode": false,
"config_rev": 3
}
}


@Balaton
now the IOMMU groups are divided
despite this

the result is this:
sudo qemu-system-ppc -M pegasos2 -bios /home/white/Scaricati/pegasos2.rom -m 1024 -serial stdio -device VGA,romfile="" -drive if=none,id=hd,file=/home/white/Scaricati/32gb.raw,format=raw -device ide-hd,drive=hd,bus=ide.1 -netdev user,id=tap0 -device rtl8139,netdev=tap0 -vga none -drive if=none,id=hd1,file=/home/white/Scaricati/coffin.raw,format=raw -device ide-hd,drive=hd1,bus=ide.1 -device vfio-pci,host=03:00.0,bus=pci.0 -device vfio-pci.host=03:00.1,bus=pci.0 -accel tcg

qemu-system-ppc: -device vfio-pci,host=03:00.0,bus=pci.0: vfio 0000:03:00.0: Could not open '/dev/vfio/20': No such file or directory

I guess there is something to fix although IOMMU now seems to work fine.

But a step forward has been made now this can be applied to any Linux distribution with Liquorix kernel.

additional information if it may be useful:
sudo dmesg | grep -i -e DMAR -e IOMMU

result:
[ 0.000000] Command line: initrd=\EFI\Pop_OS-826942db-e6c1-41bf-881f-dbe3ff07ca95\initrd.img root=UUID=826942db-e6c1-41bf-881f-dbe3ff07ca95 ro vfio-pci.ids pcie_acs_override=downstream,multifunction amd_iommu=on iommu=pt vfio.pci-ids=1002:6798,1002:aaa0
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.010008] Kernel command line: audit=0 intel_pstate=disable amd_pstate=disable initrd=\EFI\Pop_OS-826942db-e6c1-41bf-881f-dbe3ff07ca95\initrd.img root=UUID=826942db-e6c1-41bf-881f-dbe3ff07ca95 ro vfio-pci.ids pcie_acs_override=downstream,multifunction amd_iommu=on iommu=pt vfio.pci-ids=1002:6798,1002:aaa0
[ 0.586095] iommu: Default domain type: Passthrough (set via kernel command line)
[ 0.611288] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.611335] pci 0000:00:00.0: Adding to iommu group 0
[ 0.611357] pci 0000:00:01.0: Adding to iommu group 1
[ 0.611373] pci 0000:00:01.2: Adding to iommu group 2
[ 0.611391] pci 0000:00:02.0: Adding to iommu group 3
[ 0.611413] pci 0000:00:03.0: Adding to iommu group 4
[ 0.611428] pci 0000:00:03.1: Adding to iommu group 5
[ 0.611446] pci 0000:00:04.0: Adding to iommu group 6
[ 0.611463] pci 0000:00:05.0: Adding to iommu group 7
[ 0.611485] pci 0000:00:07.0: Adding to iommu group 8
[ 0.611500] pci 0000:00:07.1: Adding to iommu group 9
[ 0.611520] pci 0000:00:08.0: Adding to iommu group 10
[ 0.611536] pci 0000:00:08.1: Adding to iommu group 11
[ 0.611560] pci 0000:00:14.0: Adding to iommu group 12
[ 0.611574] pci 0000:00:14.3: Adding to iommu group 12
[ 0.611635] pci 0000:00:18.0: Adding to iommu group 13
[ 0.611649] pci 0000:00:18.1: Adding to iommu group 13
[ 0.611662] pci 0000:00:18.2: Adding to iommu group 13
[ 0.611676] pci 0000:00:18.3: Adding to iommu group 13
[ 0.611690] pci 0000:00:18.4: Adding to iommu group 13
[ 0.611705] pci 0000:00:18.5: Adding to iommu group 13
[ 0.611719] pci 0000:00:18.6: Adding to iommu group 13
[ 0.611733] pci 0000:00:18.7: Adding to iommu group 13
[ 0.611752] pci 0000:01:00.0: Adding to iommu group 14
[ 0.611769] pci 0000:01:00.1: Adding to iommu group 15
[ 0.611786] pci 0000:01:00.2: Adding to iommu group 16
[ 0.611803] pci 0000:02:00.0: Adding to iommu group 17
[ 0.611820] pci 0000:02:08.0: Adding to iommu group 18
[ 0.611837] pci 0000:02:09.0: Adding to iommu group 19
[ 0.611871] pci 0000:03:00.0: Adding to iommu group 20
[ 0.611892] pci 0000:03:00.1: Adding to iommu group 21
[ 0.611912] pci 0000:05:00.0: Adding to iommu group 22
[ 0.611934] pci 0000:06:00.0: Adding to iommu group 23
[ 0.611951] pci 0000:06:00.1: Adding to iommu group 24
[ 0.611966] pci 0000:07:00.0: Adding to iommu group 25
[ 0.611986] pci 0000:08:00.0: Adding to iommu group 26
[ 0.612005] pci 0000:08:00.1: Adding to iommu group 27
[ 0.612024] pci 0000:08:00.3: Adding to iommu group 28
[ 0.612043] pci 0000:08:00.4: Adding to iommu group 29
[ 1.612534] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002348a0]
[ 1.612653] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

ll /dev/vfio/
totale 0
drwxr-xr-x 2 root root 60 apr 5 04:17 ./
drwxr-xr-x 20 root root 4040 apr 5 04:18 ../
crw-rw-rw- 1 root root 10, 196 apr 5 04:17 vfio

lsmod | grep vfio

vfio_pci_core 94208 0
vfio_iommu_type1 45056 0
vfio 61440 2 vfio_pci_core,vfio_iommu_type1
iommufd 131072 1 vfio

------------------------------------------------------------

Next step

Using a distribution with GRUB Ubuntu+Kernel Liquorix
clean installation
is to use the commands directly in GRUB
without Aliases for the GPU

waiting for your advice