Qemu + VFIO GPU RadeonRX 550 + AmigaOS4 extremely slow

@Capehill

This is the one for science, using Radeon RX550 as the guinea pig:
https://ibb.co/mvX14CZ

@Georg
Quote:GetUpTime() results are reliable.
While it's not as easy on most PowerPC CPUs as it's on POWER and G2 CPUs (TBU = seconds, TBL = nanoseconds, overflow after 136 years) it should still be in the range of years or at least months.
Disable() may disable the TBL overflow interrupt, but not stop the 64 bit TimeBase registers from increasing monotonously.
IIRC AmigaOS 4.x uses a fraction of the FSB speed for the TimeBase register increments, but even if not and it's using something else it can be checked with the ReadEClock() return value what frequency it's using.

@nikitas
Quote:It's working correctly with the RX card, which was expected. Only with the HD card something is seriously wrong.

Something completely unrelated:
Your text output looks bad. Are you using USA or Canada as country in the Locale prefs?
If yes sub-pixel rendering may sill be disabled in the diskfont/ft2.library text rendering functions, which had to be done because of some patent issues.

@nikitas
Quote:
I think that we can safely rule out hardware issues. I've seen some threads about VFIO performance issues with Windows, and there it's always a matter of messing with the sofware configurations.


Quote:
AFAIK, that's a pretty normal set up.

Geennaam also gets poor performance with the RX 550, and his Radeon HD card also slowed down with ethernet or an emulated USB flash drive.

@balaton
Quote:
I'm shocked that the graphics card had any impact on MicroDelay(), because the graphics card is NOT involved in that function. The RadeonHD driver does have an added Disable()/Enable() when enabling/disabling the vblank interrupt, but that shouldn't be happening more than twice the monitor's refresh rate. If nothing is waiting for the vblank, then that code doesn't get run at all.

If Snoopy can log Disable()/Enable() calls, then yes, it'll be possible to see if there's more of them with the RadeonHD card.

Hans

@all

EDIT: FORGET THIS. IIRC, the drivers only use vblank interrupts. So if vblank interrupts are disabled (which nikitas has done), then there should be no interrupts.

Geennaam suggested that use of legacy interrupts might cause trouble. Modern drivers use Message-Signalled-Interrupts (MSI), and it's safe to assume that almost everyone using VFIO to pass-through their graphics card has MSI enabled drivers. We don't, because AmigaOS doesn't support MSI. So, our drivers use legacy INTx interrupts.

Legacy INTx interrupts need special handling and, given that few people will be using them, it's possible that QEmu's code for handling them hasn't been thoroughly tested. From here:

For a legacy INTx interrupt, you should know that the vfio_irq_info will report VFIO_IRQ_INFO_AUTOMASKED, meaning that the physical interrupt is automatically masked once it is triggered since these are level triggered interrupts will would otherwise continue to fire on the host until serviced by the guest or userspace driver.

To enable the interrupt, you'll want to call VFIO_DEVICE_SET_IRQS to provide an VFIO_IRQ_SET_DATA_EVENTFD for the VFIO_IRQ_SET_ACTION_TRIGGER. This will enable the interrupt and you'll be notified when an interrupt occurs via poll(2), select(2) or similar on the eventfd.

As above, when an interrupt occurs and you've finished servicing the device, you'll need to unmask the interrupt before a new interrupt can be triggered. There are multiple ways to do this. As described in the header file, you might use VFIO_IRQ_SET_DATA_NONE and VFIO_IRQ_SET_ACTION_UNMASK to effect the unmask. You could also use VFIO_IRQ_SET_DATA_BOOL, but that's a bit overkill since your bool array would contain a single index for the INTx interrupt. You can also create another eventfd, such that you have one for signaling the interrupt and one for unmasking the interrupt, in that case you'd use VFIO_IRQ_SET_DATA_EVENTFD, VFIO_IRQ_SET_ACTION_UNMASK when configuring the interrupt, and then you could simply write to the eventfd to effect the unmask.

@balaton

I tried to use Linux and Windows guest with QEMU, but I must have messed up my host system so much, both in OS and UEFI, that installations do not even start using VFIO.

When I run the qemu-system-x86_64 process, my host's monitor, which uses the integrated Intel GPU, gets crazy. This means that Windows and everything graphic turn to a strange multicolour pattern as if the integrated GPU is malfunctioning. I see a "flash" on the guest's monitor, but it doesn't start booting.

When I stop the QEMU process, the host's monitor turns back to normal, and the artifacts are gone.

I have to reset UEFI settings and some of the OS settings and try again.

@Hans

I barely understand a thing. But does this mean that we have a clue now?

When I start the qemu-system-ppc process it turns the card to the legacy IRQ mode INTx:


When I stop the process, I see no difference. But when I reboot the host system, it gets back to (which means MSI?):

@Hans
The QEMU vfio device should do all that managing interrupts, from the guest it should appear as a normal PCI interrupt. There could be problems however if there is a problem somewhere passing through the interrupts. But I think if other PCI interrupts work, the vfio-pci device uses the same way so it should work too on the PCI side of it. I don't know about the vfio side. We're also passing a PCIe card as PCI to the guest so it may be an unusual config that's not well tested. As long as the guest does not try to access PCIe parts it might work though. There are some traces in QEMU's vfio that could be enabled, look in qemu/hw/vfio/trace-events. You can enable all of them with -trace enable="vfio*" but that might generate a lot of logs. Some logs might also be in the Linux kernel logs as vfio is handled there. Maybe there are some clues that could be get from those?

@HansQuote:

It's not known if it's MicroDelay() specifically or if "with that gfx card" any other task which does something (heavy, like some other kind of benchmark, or calculation, or even just a compilation of some code) would see the same slowdown.

@Georg
Guest-only code like compiling something may not have such extreme slow-downs, no matter what the reason for the slow speed is.
MicroDelay() reads the emulated TBU/TBL registers, which probably requires exiting the JITed PPC guest code, executing host QEmu code to set/update the emulated TBU/TBL registers and returning back to the guest code.
I don't know how QEmu works, but limiting QEmu to a single host CPU core with taskset may cause additional slowdowns in such cases.

@balaton

I don't know if this helps at all, but I've uploaded vfio_* trace logs from the time of guest boot until the GPU test 16.


https://ufile.io/yf28ui39 (30Mb)

PS: I have passed through only the VGA controller (00:01:00.0) of my GPU, as the Audio Device (00:01:00.1) is not even recognized(?) by AmigaOS4.

Is the slowness still there if you pass through slow gfx card without actually using it in AOS. Maybe you need to move it somewhere from it's directory "(DEVS:Monitors"?)to prevent it from being loaded.

If not, what if you then start gfx driver manually (double click driver icon?).

What, if you then change wb screenmode to use that gfx card.

What, if you then change back screenmode to something (not that gfx card) else again.

@nikitas
I don't see anything extraordinary in the log (but I also don't really know what to look for). It seems using legacy interrupts may disable mapping of BARs so it may use slower I/O to access those but that still does not explain the slow speed you got also when others already manged to get better results.

Did you try other things that was mentioned above such as not using taskset and remove cpu isolation and irq pinning (or at least allocate 3 CPU cores to QEMU)? Or the tests Georg has described both with x11perf and in the last post above assigning the card but not loading the driver and see at which point it becomes slow (just to locate which component does something that could make is slow).

@nikitas

I have rebuilt the Linux kernel "Longterm: 4.19.319 2024-07-27" for Pegasos 2.
If you need it, let me know.