Qemu + VFIO GPU RadeonRX 550 + AmigaOS4 extremely slow

Hello all,

CC: @Hans @balaton

RadeonRX 550 (my secondary GPU) is recognized by Qemu 9.0.50, PegasosII & AmigaOS4.1 FE Upd 2.
To make RadeonRX 550 work, I had to go to host's BIOS and disable the "Resize BAR" setting.
In AmigaOS4, RadeonRX 550 seems to work, and two new processes appeared: rx_gc, rx_pm.

The problem is that AmigaOS is extremely slow - while without VFIO, using only SM501, it's much much faster.
During the guest BIOS boot I get the message:
System doesn't have full memory coherence. Compensating...

But the lowest CPU consumption is 61%, and this only when it is idle.
When I move the cursor or open a folder, CPU is always 100%.


Some images from Ranger:
https://ibb.co/dpMRtp1
https://ibb.co/wCqnXH2
https://ibb.co/qnhWZ23
https://ibb.co/F8NP25b


Is there a way to fix this performance issue?


Below, you can see the specifications of my systems (host, guest, emulator) and the corresponding configurations.


Host System:
CPU: Intel Raptor Lake-S (Core i5) 13th Gen Intel(R) Core(TM) i5-13400
CPU Instructions: HT, MMX, SSE(1, 2, 3, 3S, 4.1, 4.2), AVX(1, 2), FMA(3), AES, CLMUL, RdRand, SHA, VT-x, x86-64

Motherboard: ASUSTeK COMPUTER INC. PRIME Z790-P D4 Rev 1.xx
BIOS Version: American Megatrends Inc. 1656 04/18/2024

Operating System: Ubuntu 24.04 LTS
Linux Kernel: Linux 6.8.0-31-generic

Graphics Card Driver: vfio-pci
Graphics Card Model: PCIe Sapphire RadeonRX 550 --> Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X]


lspci -nnk Results:



Emulation System & Configuration:
- QEMU emulator version 9.0.50
- Bboot v0.7

Qemu command:


Guest System:
- AmigaOS4.1 FE Update 2
- Enchanched Software v2.2
- RadeonRX.chip v2.12


Kickstart Layout with RadeonRX enabled:




Boot logs with RadeonRX.chip.debug enabled:

Hello Nikitas,
Welcome to our forum.
I do not know much of what you are doing, but have you checked the https://www.amigans.net/modules/newbb/viewtopic.php?topic_id=9568 ?

Balaton mentions there that the RX550 is slow for him as well.
This is a topic I want to investigate more myself, so I am looking at these discussions with increased interest.

@walkero

Hi there,

What I am trying to do is a passthrough of my secondary host GPU, a Sapphire RadeonRX 550 on Qemu, PegasosII emulation machine. On this QEMU setup, I installed AmigaOS4.1 FE Pegasos II Edition. Then with great help from @balaton via email, I managed to make a successful VFIO passthrough.

Before that, I had to isolate my RadeonRX GPU on the host system so it only uses the VFIO driver.

I plugged in the secondary card on a second monitor. Using an HDMI-to-HDMI cable. It seems to work because AmigaOS reports that is using the card via the RadeonRX.chip driver. At this moment my problem is that AOS4 is extremely slow. So slow that you have to be very very patient to do the slightest operation, such as double clicking on a Workbench folder... The window appears after 2-3 seconds and I can see it drawing and drawing for another 5-6 seconds.


It is true, that a real PegasosII does not have a PCIe slot. Only PCI. My Radeon GPU is plugged into a PCIe slot because my motherboard is relatively new, so it does not have a PCI slot. I don't know if this contributes to the slow performance.

I don't even know if the problem is on QEMU side or on AmigaOS4 Radeon.chip side. But I guess that the problem may exist in the AmigaOS part.

On the post that you've sent, Zoltan may refer to the conversation we had via email. Not sure.

@nikitas

Just to check - have you gone into Power prefs and changed the gfx card settings to fastest possible?

@derfs

Yep, I've done this already...

@nikitas
Quote:No, the fast RadeonHD cards used are PCIe as well.

Since you get 100% CPU usage even for very simple things using a tool like http://os4depot.net/?function=showfil ... ity/workbench/tequila.lha or my top may help to check which task(s) are using the CPU.

For the RadeonHD cards with fast results the Pegasos2 firmware was used in addition to BBoot, you could try if that changes anything (may require additional steps, something like "/failsafe" in the firmware prompt, but I don't know anything about the Pegasos2 firmware).

@joerg

Here you go,

https://ibb.co/1bVsZ31

.....
idle. Task 64.1%
compose. Task 10%
page_sweep 2.8%
.....



Also, I have tried both ways. Using pegasos.rom and bboot. Both work, the same slow performance though.

@nikitas
Yes, I've quoted your report and log in the other thread for RX 550. What happens if you remove bus=pci.0 from the vfio-pci devices? Does it help if you disable compositing or select a theme that does not use transparency? (Just because compose.task shows up using CPU in your screen shot but I don't know what that does ot if it's related.) The problem can not only be in QEMU and AmigaOS but also on the host side of things. Does this card have UEFI BIOS or does your UEFI have CSM enabled in setup? Do you have another machine to try this card with to see if you get the same problem to rule out problem with the host?

@balaton

Hello,

When I remove the bus=pci.0 it gets stuck with:



By using the bus=pci.0, it still shows this message above but it continues and boots.

I have tried with both CSM Disabled/Enabled.
In case of CSM Disabled I've tried both PCI "Legacy Only" and "UEFI Only" modes. The latter was unnecessary to try, I think.

Also, in UEFI, by having the "Resize Bar" setting to "Enabled", the above message is printed again, not once but periodically, every half of a second, and the system does not boot.

I disabled compositing effects on AmigaOS, and any transparency setting I found, but no luck. It is slightly more responsive but not sustainable for someone to use.

I currently do not have a second machine to try the GPU.

FYI: I use the same card with VFIO, virsh, to run a Windows 11 guest, and it works smoothly.

@nikitas
Quote:
So that message may not be related to this and is shown anyway. Something is probably trying to print a diag code as this register is used for diagnostics on ISA bus but if it's just one char it's probably not something we can interpret without knowing what prints it. Do you get this log even without -bios pegasos2.rom and do you only get it when you have the passed through GPU or even without the -device vfio-pci options?

Maybe there's not enough space to map GPU BARs on the default PCI bus. What does info pci show in QEMU monitor with and without bus=pci.0 after AmigaOS started to boot or as far as it gets?

Quote:
When it prints it periodically you said before it may try to reboot again and again. You could add -d unimp,guest_errors to QEMU command to get some more logs and maybe that also shows if it reboots or not (if you get all logs periodically then it's likely a reboot otherwise it's just stuck in whatever causes this write to port 80).

Quote:
OK if that's the same machine, same card, same QEMU then the host side at least should be OK. Does it also work with a Linux guest instead of Windows? Does it work with a PPC Linux guest on pegasos2? If so then problem is most likely in AmigaOS side otherwise we should be able to reproduce with PPC Linux guest which may give better logs to find the cause of the problem.

@balaton
Quote:compose.task does the alpha blending with transparency, adding shadows, etc. of the window bitmaps to the screen bitmap.
It's using the 3D features of the GPU and therefore doesn't copy much data around (except on GPUs without 3D support like SM501/2 where CPU based workarounds have to be used), but just sends GPU commands to the gfx card.
The only reason I can think of that that's very slow would be extremely slow access to the gfx card VRAM.
Maybe a general problem with VFIO on QEmu?
For example

https://lists.hdrlab.org.nz/benchmark/ ... 2d/OS/AmigaOS/Result/2348 (X5000/20, Radeon RX Polaris10)
Operation MiB/s
Copy to VRAM 533.72
Write Pixel Array 1,071.60
Copy from VRAM 40.33
Read Pixel Array 995.02

https://lists.hdrlab.org.nz/benchmark/ ... 2d/OS/AmigaOS/Result/1551 (X1000, Radeon HD 6970)
Operation MiB/s
Copy to VRAM 438.44
Write Pixel Array 1,415.46
Copy from VRAM 41.77
Read Pixel Array 398.56

https://www.hdrlab.org.nz/benchmark/gf ... 2d/OS/AmigaOS/Result/2660 (Sam460ex, Radeon HD 5450)
Operation MiB/s
Copy to VRAM 305.30
Write Pixel Array 540.02
Copy from VRAM 59.13
Read Pixel Array 60.30

https://www.hdrlab.org.nz/benchmark/gf ... 2d/OS/AmigaOS/Result/2773 (QEmu on i5, Radeon HD 7800 Series (MSI R9 270x))
Operation MiB/s
Copy to VRAM 67.15
Write Pixel Array 65.74
Copy from VRAM 6.09
Read Pixel Array 6.16

@joerg

Ouch! Those VRAM access stats are horrifically bad, and could indeed explain poor performance. It's a double-whammy:

First, the driver doesn't use GART on older systems like the pegasos-2, A1-XE, & Sam460 due to lack of memory coherency. I tried manual cache flushing/invalidation, but simply couldn't get it to work reliably. So, all commands are written to VRAM. Therefore, slow VRAM access means slower command submission.

Second, AmigaOS does a lot of CPU-based rendering. Far too much. This includes things such as drawing text and lines (other than horizontal or vertical). This will frequently involve both reading from and writing to VRAM. Reading from VRAM at just 6MiB/s is especially bad. A 32-bit 1080p screen is ~8 MiB, which would take approx. 1.33s to read...

Disabling text anti-aliasing should speed up rendering of text. I bet the whole system would still be sluggish, though.

The graphics library does use DMA to copy to/from VRAM on some platforms (e.g., the Sam4x0, X1000, X5000, & possibly A1222). Emulating one of those platforms might help, provided that emulation of the CPU's DMA engines uses actual DMA. PCI-Express busses have a high penalty for accessing small blocks of data (e.g., just 32 or 64 bits).

Hans

@Hans
Quote:And that was even the fastest result of a passed-through gfx card on QEmu until now, other results are even worse, for example
https://ftp.hdrlab.org.nz/benchmark/gf ... 2d/OS/AmigaOS/Result/2772
Operation MiB/s
Copy to VRAM 4.19
Write Pixel Array 4.38
Copy from VRAM 2.24
Read Pixel Array 2.26

@Hans @joerg

I ran this test too. It took about 3 hours to finish.

----
Operation MiB/s
Copy to VRAM 98.28
Write Pixel Array 101.30
Copy from VRAM 4.29
Read Pixel Array 4.29
----

http://hdrlab.org.nz/benchmark/gfxbench2d/OS/AmigaOS/Result/2784 (Pegasos II/MPC 7447/7457, Radeon RX Polaris12)