QEMU GPU vfio-pci pass through

The system I'm using has a Ryzen 5 5600x, and two GPUs:

XFX Radeon RX 580
Sapphire Radeon R9 270X

I installed ArchLinux to use Qemu 9.2.0 and set up GPU passthrough with VFIO. While trying to get the passthrough working, I wanted to use R9 270x with Amiga side. The R9 270x video and audio components were tied up in IOMMU groups with other devices and QEMU didn't like that, so I then implemented ACS write patch to break out each device into its own group (using linux-zen kernel).

I never got it to work, so I swapped to using the RX580 (by changing the vfio ids in the grub line, running mkinitcpio again) and I was able to finally see the Amiga boot on it's own monitor, but now the mouse is frozen in the upper left. I tried the bochs-display to capture mouse events but there was an error ... can't reproduce it now but mouse still frozen. Will next try to pass through a specific mouse device on the command line.

@themamboman

Could you link me the model of yours (the one you are using) XFX Radeon RX 580
on Amazon so I understand how much I have to spend to get it and the exact model.

I recently got the rtx 4060
so I have rtx 2060 aside that I wanted to keep but I can sell it.
And eventually get the XFX Radeon RX 580.

Don't worry I'm not a person attached to money but I would like to understand which is the exact model to buy for qemu.

also I have the version of "Enhancer Software 2.2 for AmigaOS 4.1"
are the drivers included in that package ?

What I would like to use is AmigaOS 4.1 as a system and system acceleration for video and applications in general. (e-uae)
I am not interested in games that part is not important to me.


Thanks.

Today I backed up my system.
I had configured "VSphere" some time ago but I needed a second graphics card to use it and so I did my tests on this.
The discussion is broad and I will not dwell on this.

Also I was using the new:
https://www.techpowerup.com/download/nvidia-dlss-dll/
version (310.2) for DLSS.
( to enable dlss on some games (not multiplayer could lead to a "ban" because it is seen as tampering with the original game )

Now I use an ASUS TUF GAMING B550 with the Ryzen 5800 with the latest updated bios (3611).
I have 2 graphics cards 4060 and 2060

And I was doing some tests with some settings in the bios

PCI Subsystem
4g Decoding ENABLE
Resize BAR ENABLE
SR-IOV Support ENABLE

with these settings enabled in BIOS

4g Decoding ENABLE
Resize BAR ENABLE

the usb stick could not boot for the backup.
I have to see if this also happens with the "mouse".

Setting both to DISABLE the USB stick would boot normally again.

I don't know if you have these settings on your motherboard.

Let me know what card you use as soon as you can also a link on ebay to understand the model to buy.

Thanks.

Model number RX-580P8DFD6

https://www.amazon.com/XFX-Radeon-1386 ... -580P8DFD6/dp/B06Y66K3XD/

@themamboman

Thanks
it's around 100 euros used ( it's equivalent to a geforce1060 )
release date around 2017.
I'll post the video the model you indicated should be this one:

https://www.youtube.com/watch?v=C-loUscDLdw

at minute 2:17 you can see the label


note:
Keep me posted on your tests.
I think I'll buy it at the end of the month.

Thanks

here is the problem that occurs using:
PCI Subsystem
4g Decoding ENABLE
Resize BAR ENABLE

when I connect the USB stick to use in this case "TrueImage"

if I disable it:
PCI Subsystem
4g Decoding DISABLE
Resize BAR DISABLE

the problem does not occur and the boot proceeds without problems.






In subsequent versions Acronis has solved the problem starting from the "2022" version in my possession.
All versions are regularly purchased and therefore original.

the power supply is this:
Corsair RM750x 750 Watt with 80 PLUS® Gold certification

in fact with the 4060 the consumption has dropped and it works at PCIe 8x:

worst case scenario I'll resell it if it doesn't work with qemu

I have a somewhat stupid question to ask,

At the moment I only have one monitor
AOC 27G2U/BK 27 Full-HD Gaming Monitor (1 ms, 144 Hz)
here is the photo taken on the internet to understand better

Do I need a second Monitor?
Because at the moment I only have this one.
In case a TV with HDMI I imagine it should be fine.

Because I have never done a configuration with 2 cards
So I install native Linux with basic NVidia drivers
and then with qemu I do the Passthrough directly pointing to the Radeon 580 card ?

Clearly I do not need a step by step guide.
But just understand the principle of how I have to set up the machine on the hardware side.

Thanks for any suggestions.

@themamboman
Maybe you can try adding -device usb-mouse with bochs-display and see if that works better. If not you have two options, either use USB pass through with -device usb-host,some-more=parameters-here to pass one USB device but then you likely need to add a udev rule in /etc/udev/rules.d to tell the host to not use this device so it can be used by the guest and won't control your host mouse. I don't know what to put in such udev rule but there should be some docs online on how to pass a USB device with -device usb-host. Or you could add a card with USB ports and pass that through with vfio the same way as the GPU so anything you plug into that card would be controlled by the guest. For that you may need such USB card supported by AmigaOS and a free slot that can be used with vfio for this.

@Balaton
do you have any advice for the questions I asked about setting up the hardware ?

Thanks.

@themamboman

Here you can find some tips on how to create udev rules for an audio usb stick and other tips if they are useful.
I also made a video I must have it somewhere if you need it.

https://eab.abime.net/showthread.php?t=106392&page=5

@white
I have no time to help you setting this up. I should write some docs on this but I also don't have time for that. But here are some quick point for consideration.

- If you don't have Enhancer you can maybe only use older RadeonHD cards without 3D. 3D drivers and RadeonRX are in Enhancer so don't bother buying such cards unless you have Enhancer. For first experimenting a RadeonHD card may do which are very cheap.
- You may get problems if using on-board Intel GPU as these don't seem to work well with the VGA part so you may need two cards in this case and disable Intel GPU otherwise the BIOS of the passed through card can't access the VGA registers as these are held by the Intel GPU and can't correctly init the card so it won't work.
- If you have a new card with gigabytes of RAM that may need the 4G and resize BAR options as you've found but this may be a problem with AmigaOS which may need the opposite setting so in this case you may need another machine with two older cards or another older card but you may not want to downgrade your GPU for this.
- You don't need more monitors, you can connect the second card to another input of your monitor then switch between inputs on the monitor. It's not very convenient but enough for testing.
- There are probably more points in this and other threads I forgot by now.

@balaton
Thanks,
Perfect

I don't have integrated graphics cards, the ryzen 5800x doesn't have them.
And I would like to use Linux with Nvidia drivers (rtx 4060)
as the main system.

My motherboard supports 2 graphics cards

and then with qemu I try to do the passthrough on the radeon 580 gpu ?

I have the 2.2 package of Enhancer

In summary I use Linux as I always have with Nvidia drivers (4060)
and then via qemu I passthrough to the 580 mounted on the same motherboard

Thanks



here i found this topic in amigans:
https://www.amigans.net/modules/newbb/viewtopic.php?topic_id=9719

@themamboman
How are the tests going ?
Were you able to get some improvements ?
Thanks

Together with this you should also disable CSM as suggested in the bios to resize BAR.
It seems to work at 64bit so I don't know if it's useful for emulation

Still on the Resize BAR I found this Linux side Radeon 580:

https://forums.guru3d.com/threads/perf ... s-vega-radeon-vii.445141/

@themamboman
Can you provide me with some useful links to find the changes you made ?

I was looking at some stuff:

https://github.com/benbaker76/linux-acs-override?tab=readme-ov-file

https://forum.level1techs.com/t/the-pr ... fio-iommu-and-pcie/111251


It would be interesting to see the syntax you use with (qemu) to initialize the Radeon 580 this would be helpful for me.
an example of command line.
Thanks

I plan to make a post about my efforts if they ever work out on my blog: amigafun.wordpress.com

Okay, so balaton's suggestion of adding -device usb-mouse and -device bochs-display seemed to work as far as mouse movement was concerned. The downside is that in order to capture the mouse movement, I have to only use it within a very small black window that pops up over the linux desktop. if I move the mouse outside that little box, the mouse pointer on the amiga monitor freezes until I get it back within the boundaries of that box.

When it did work, the system was VERY, VERY SLOW. I think this is something found in other threads so I haven't had a chance to read up on their solutions (if there were any).

I also went through so many changes that I'm not sure about making a step-by-step tutorial. I basically used the notes balaton made on this form as well as his Amiga-like OS's webpage with emphasis on the pegasos2 page.

Basics:

- Installed the latest archlinux on my AMD system (choose grub as bootloader)
- Followed VFIO and GPU passthrough tutorials on KSK Royal's YouTube page (turning on virtualization and IOMMU as well)
- Follow how he found the radeon vendor and model ids are added them to grub and regenerated grub bootloder.
_ He used Nvidia card, so I had to change the file /etc/modprobe.d/vfio.conf to use the vendor:model numbers as well as change second line to be `softdep amdgpu pre: vfio vfio-pci
- Had to make a new file in same folder vfio.conf called amdgpu with the only line being the same as the 2nd line of vfio.conf
- arch doesn't regen modprobe changes like ubuntu in his video so had to use one for arch: sudo mkinitcpio -P
- then had to install linux-zen using pacman -Sy linux-zen.
- added `pcie_acs_override=downstream,multifunction vmlinux=zen` to end of grub GRUB_CMDLINE_LINUX_DEFAULT line after vfio additions.

I had to use acs overwrite to break each pcie device into it's own IOMMU group. I had to do this because the 270x card was showing up with several other devices.

The command line was something like this:



Oh and I used WinUAE (Flowerpot) running AmigaOS4.1fe Classic to log into AmiStore and purchase/download Enhancer 2.2 to get the RadeonHD and RadeonRX chip files to include in the Kickstart.zip file. It was much cheaper buying it there than physical media from a vendor.

I have also noticed that I can only run the qemu command a few times before I get this error:

qemu-system-ppc: ../qemu-9.2.0/hw/pci/pcie.c:1024: pcie_find_capability_list: Assertion `next <= PCIE_CONFIG_SPACE_SIZE - 8' failed.

Only rebooting the computer will allow it to work again.

@themamboman

Thanks for the info,
So you haven't had time to test the XFX Radeon RX 580 8gb yet ?

In the meantime I downloaded (Linux-Garuda) linux-zen kernel

Even though last time I used (POP!_OS) System76 with the Nvidia drivers already pre-installed in the iso.

and I have to say it was much faster than Kali.

When you have time let me know if you've tried the XFX Radeon RX 580 8gb ?

Because it's the one I'd like to buy but there's still a week to go before I buy it.

Thanks.

note:
if it doesn't work I'll resell it and buy another 16gb of Corsair ddr4 3600 ram

So with 32gb of ram I can do various virtualizations as I've always liked
--------------------------

It's not bad here it's on virtual machine I wanted to try it
I will use this distribution I don't need Linux I need the emulation of 4.1 (if possible)





I also leave this link on YT it's in Italian as a reminder.
But the difficult thing is to do it with (AmigaOS) obviously.
This person is really good at handling linux (just watch his other videos).
Maybe I'll send him an email.

https://www.youtube.com/watch?v=gpJ5eZK3PfI

BIOS settings for IOMMU:

SVM -> Enable (OK)
iommu -> Enable (OK) (AUTO)
ACS ENABLE -> Enable (OK)
Enable AER cap -> Enable (OK)
Alternative routing -> Enable (OK)