You are basing your analysis on unproven assumptions.
Wrong, I was one of the main AmigaOS4 developers and know how it works. Although I wasn't involved since about 15 year in it any more some things are impossible to implement in a completely different way. Unless you remove compatibility to ancient AmigaOS 1.x-3.x/m68k software, which I'd have preferred, but nearly all other OS4 developers preferred compatibility to AmigaOS 1.x-3.x software over better implementations only usable in new AmigaOS 4.x/PPC software. I very much doubt anything has changed in that regard, and there are next to no competent developers left still working with Hyperion.
Quote:
Without knowledge on how AmigaOS accesses the card I'm not sure yet the problem is really because of fine grained access to VRAM so this is still something to verify.
Maybe you don't know, but I do
Quote:
I'm also not sure the PPC440 DMA engine on sam460ex is used for this.
I am. If you don't trust me simply ask @m3x from ACube, he implemented several Sam440/460 AmigaOS 4.x parts, incl. the DMA improved CopyMem(Quick)()/bcopy()/memcpy() (used for graphics.library (Read|Write)PixelArray() and several other AmigaOS 4.x parts as well), etc.
@joerg Do you also know how the Radeon RX driver works? I thought only @Hans knows that as he keeps the sources so nobody else knows. If the issue is that other parts of AmigaOS acesses VRAM directly which is slow on these cards then this would also be a problem on real hardware. How is that solved? Maybe it does something that works on real machine to avoid this problem which does not work well with QEMU but since I don't know what it does I also can't fix it. Maybe @Hans could make a test case that is doing VRAM access the way the OS or driver does in a small C program that can be compiled and tested so we can check what host code it is compiled to by QEMU and do some tests with that. Otherwise it would be difficult to fix. It's not reasonable for QEMU to emulate a card on top of vfio to add DMA to VRAM access that the OS or driver should do in the first place.
If this is the problem then maybe looking for other cards to pass through which are faster for this could be a solution. Some cards support unified memory or are integrated graphics cards where accessing the VRAM should not be a problem so maybe those could work better? It should be possible to pass through even an integrated GPU but then you either need another card for the host or live with the host graphics shutting down when you run QEMU (or only have output from host via serial). I've found this guide vfio-single-amdgpu-passthrough that shows something like that. It also mentions at the end that AMD cards have some reset issues that may need an external Linux kernel module to fix which may be useful for some trying to pass these cards through.
A minor update. When you disable compositing effects and downgrading resolution to 16 bits, 64k colors, then "Enable Interrupts" in "Screen Mode" works (as slow as without it).
While the workbench windows are struggling to draw, when I use Amistore (written in Hollywood, as I read) it's very fast. I login, navigate through the menu, no visible drawing or delays. Here, using RadeonRX has the "same" speed as the sm501 emulated card. (did not measured anything to say precisely, mentioning only as a user experience).
I don't know if this has to do with anything, just reporting it.
@LiveForIt GART only helps if the GPU does all rendering and the copies between VRAM and DRAM. That's the case on all other OSes, except for AROS and MorphOS maybe. On AmigaOS for example Warp3D may benefit from GART, but not the 2D graphics.library where the CPU does a lot of the rendering and all copies between VRAM and DRAM.
@balaton Quote:
If the issue is that other parts of AmigaOS acesses VRAM directly which is slow on these cards then this would also be a problem on real hardware. How is that solved?
On the old systems (G2 classic Amiga, G3+G4 AmigaOne/Pegasos2) not at all, it's not possible. VRAM copies are tiny, slow 64 bit (G2+G3) or 128 bit (G4) CPU accesses as well. PPC CPUs seem to be faster accessing VRAM over ZorroIII/PCI/PCIe than x64 CPUs are, but not very much. On the newer NG systems, Sam4x0, X1000, X5000 and A1222, the CPU DMA engines are used for the copies between VRAM and DRAM which is much faster because of the much larger parts copied at a time (= much less PCI(e) overhead).
Quote:
Some cards support unified memory or are integrated graphics cards where accessing the VRAM should not be a problem so maybe those could work better?
There are AMD x64 CPUs with integrated Radeon gfx, mostly used for laptops, but I don't know if there is any on which the gfx part is compatible with the AmigaOS Radeon HD or RX drivers.
With a bit of luck, you can buy ATI cards suitable for testing for a few euros in China. Since Vladimir Vladimirovič is practically selling off his homeland to China. I foresee Chinese restaurant chains in Vladimir Vladimirovič country. Even oil is sold at controlled prices to Chinese residents in their homeland. This is simple finance. On the other hand, how can you blame China? This is the biggest deal they've had in many years to the detriment of their neighbors led by Vladimir Vladimirovič.
hey, a little humor doesn't hurt every now and then.
I hope a suitable card for pass through will be found.
I am currently conducting a Special Tracing Operation in order to invade QEMU PPC translation blocks and see what is running differently with and without VFIO. Sadly, I think it will take the same or even longer than the "Special Military Operation".
Using qemu tracing on specific trace-events would be useful in our case? In order too see what TBs are running with & without VFIO. If yes, what events need to be traced?
I'm trying to study some QEMU tracing backends but I'm still missing knowledge to understand what exactly have to do. Anyway, I'm eager to learn QEMU internals as much as I am capable of, of course.
Also, currently, I passed through a real HD drive for my QEMU PegasosII AOS4 VM. I don't know if this can have any positive/negative impact on performance.
Regarding the RadeonRX metrics needed by @Hans, I'm consciously avoiding it, because it is a little boring to De-Vfio the GPU, remove amdgpu blacklist and any other thing might needed.. And then VFIO the GPU again etc. But if it is going to help, I'll have to do it.
You also insisted on using pci.1 instead of pci.0. Indeed, I told you that with VFIO, the pci.1 GPU did not work for me... Pegasos loads the Kickstart and then gets stuck forever.
Has anybody managed to pass through the RX GPU via pci.1 (PegasosII machine)?
Edited by nikitas on 2024/6/18 21:41:31 Edited by nikitas on 2024/6/19 4:12:28
@joerg Do you also know how the Radeon RX driver works? I thought only @Hans knows that as he keeps the sources so nobody else knows.
A-EON owns the driver, so the source code is on their repository. Multiple people have access to it, although it's rare for anyone other than me to touch the code.
The driver will use GART for sending command buffers to the GPU, and also for transfers to/from data buffers used by 3D drivers. Sadly, the Picasso96 API doesn't allow the driver to do the same for the graphics.library.
This is only for motherboards/CPUs that are confirmed to have working memory coherency: X1000, X50x0, A1222. The driver falls back to CPU-based copies to/from VRAM for all other motherboards.
Quote:
If the issue is that other parts of AmigaOS acesses VRAM directly which is slow on these cards then this would also be a problem on real hardware. How is that solved? Maybe it does something that works on real machine to avoid this problem which does not work well with QEMU but since I don't know what it does I also can't fix it.
The graphics.library will use the CPU's DMA engines for copies on platforms where it has those routines (e.g., Sam 4x0, X1000, X50x0, A1222). If that's not available, then it has copy routines that are optimmized on a per-CPU basis. So, it'll use altivec based copy routines on CPUs with altivec, and 64-bit float based routines elsewhere. It'll even use the A1222's SPE.
One trick that these copy routines do, is to ensure that the VRAM side is always correctly aligned. Mis-aligned accesses are expensive. and don't even work on some platforms.
Just in case anyone missed it: yes, the copy routines use 64-bit and even 128-bit (altivec) transfers to maximize the copy speed.
Quote:
Maybe @Hans could make a test case that is doing VRAM access the way the OS or driver does in a small C program that can be compiled and tested so we can check what host code it is compiled to by QEMU and do some tests with that.
I already have: it's called GfxBench2d. It includes some of the same copy routines used by the graphics.library and RadeonHD/RX. It benchmarks multiple copy algorithms, although they're NOT all uploaded to the HDRLab website. Check the text output.
Quote:
Otherwise it would be difficult to fix. It's not reasonable for QEMU to emulate a card on top of vfio to add DMA to VRAM access that the OS or driver should do in the first place.
Yeah, that would be ridiculous. My top questions are: - What's the VFIO performance when running Linux (PowerPC) inside QEMU? Do they also have a big performance hit? - What does the generated host code look like? If it isn't a tight loop, then the CPU/PCIe-controller won't be able to batch up transfers into larger packets - How does VFIO even work? Can the generated host code access the hardware directly? Or is there some kind of abstraction layer in-between. If there is a layer in-between, what's its overhead? For example, do accesses run through an exception handler?
Huh? What on earth is going wrong there? Why is your card's FillRect ops/s maxing out at a level roughly 368x slower than his? That is a huge drop in performance.
I don't think that the memory copy results can explain that. Your RAM=>VRAM copy speed is actually ~49% faster, and your VRAM=>RAM speed is ~43% slower.
Hm.. Wait a minute. I have another VM with Windows 11 Guest in which I have VFIO'd exactly the same GPU card and I downloaded there your Windows version of Gfx2Bench2D. Your tool is a bit outdated compared to Amiga version but it produced the following results:
The Amiga version (on QEMU AOS4.1) has 54 tests and takes 3 hours. The Windows version (on QEMU/KVM Windows 11) has 38 tests but takes... seconds to finish.
I don't know if this is a checkmate, but the question remains for me. The results are showing: Better QEMU VFIO handling on x64? Or better handling of the Windows AMD Driver?
------------------------------------------------------------
GfxBench2D 2.9
A benchmark tool for graphics cards.
Written by Hans de Ruiter.
Copyright (C) 2011, by Hans de Ruiter, all rights reserved
------------------------------------------------------------
System Information:
OS: Windows 11 Home (build 22631)
Motherboard/Device: Standard PC (Q35 + ICH9, 2009), manufactured by: QEMU
CPU: 13th Gen Intel(R) Core(TM) i5-13400, @ 2.496 GHz
L1 Cache Size: 65536, L2 Cache Size: 16777216, L3 Cache Size: 16777216
Total RAM: 15.9808 GiB
External Bus (FSB) Speed: 0 Hz
Clock granularity is: 1030us
Tests will take at least 1.030 seconds each.
Hm.. Wait a minute. I have another VM with Windows 11 Guest in which I have VFIO'd exactly the same GPU card and I downloaded there your Windows version of Gfx2Bench2D. Your tool is a bit outdated compared to Amiga version but it produced the following results:
The Amiga version (on QEMU AOS4.1) has 54 tests and takes 3 hours. The Windows version (on QEMU/KVM Windows 11) has 38 tests but takes... seconds to finish.
I don't know if this is a checkmate, but the question remains for me. The results are showing: Better QEMU VFIO handling on x64? Or better handling of the Windows AMD Driver?
The drivers and graphics APIs are definitely a lot better. I don't know if QEMU VFIO handling is better with x64 client OSes, though.
GfxBench2D can't test CPU RAM<=>VRAM transfers on Windows, because the API doesn't allow it. The WritePixelArray/ReadPixelArray results are DMA accelerated.
The driver is managing 2896694.319 ops/s with FillRect 16x16, which is over 28900x faster than what you're getting with AmigaOS (and about 77x Geennaam's results).
The Windows drivers have multiple advantages. The modern Windows graphics APIs have the concept of command queues. So, it can send draw operations in batches for maximum performance. AmigaOS' graphics.library API draws everything immediately (so, no queue). This means that every draw operation is sent to the GPU individually, which is the worst thing to do performance-wise. We're continually hitting the bottleneck of how many command-buffers/s (or batches/s) can be handled (because we're submitting batches of one).
So, I'd expect Windows & Linux drivers to outperform our drivers just based on the batching.
I've always wanted to batch the draw ops, but see no safe way of doing so with the graphics.library, because it has no Flush() or Finish() function to flush the queue (i.e., apps/games expect draw ops to happen immediately, and never signal a flush/finish).
There's still something going wrong on your system vs Geennaam's. His system has a bottleneck of about 37500 command-buffers/s, whereas yours is maxing out at just 100 command-buffers/s. Being able to draw max 100 things per second is painfully slow.
@afxgroup Even if it's in a laptop it could be a discrete GPU not using the same memory as the CPU so it might not be different from a GPU on a graphics card but it would be interesting to see the results anyway. Setting up pass through on a laptop might be more difficult though because you would lose graphics output on the host while the guest is running and it should be scripted to get it set up and tear down vfio on guest start stop to work. You probably will need serial or ssh access to the host while experimenting setting it up as the screen may not work and will only display the guest.
Using qemu tracing on specific trace-events would be useful in our case? In order too see what TBs are running with & without VFIO. If yes, what events need to be traced?
It would be difficult to identify the interesting code section in the lot of code the OS runs otherwise. You can enable some logs, see -d help but I don't see how to identify the part that's relevant for this unless we can cut the code down. See reply to @Hans on this below. Quote:
I'm trying to study some QEMU tracing backends but I'm still missing knowledge to understand what exactly have to do. Anyway, I'm eager to learn QEMU internals as much as I am capable of, of course.
The tracing backends are only determine the way the logs are saved, the 'log' backend just prints them to stdout/stderr which can be easily redirected so for simple debugging it is usually enough. Other backends can save to binary files or send to syslog. What is logged is determined by the -d options independent of the log backend. Quote:
Also, currently, I passed through a real HD drive for my QEMU PegasosII AOS4 VM. I don't know if this can have any positive/negative impact on performance.
You can measure it to find out. There were some file system benchmarks for AmigaOS I think. My guess is that for small files (like normal OS usage) it may not be a big difference as the IDE overhead would be bigger. The host filesystem overhead is probably not too big compared to the guest side overhead. Quote:
Regarding the RadeonRX metrics needed by @Hans, I'm consciously avoiding it, because it is a little boring to De-Vfio the GPU, remove amdgpu blacklist and any other thing might needed.. And then VFIO the GPU again etc. But if it is going to help, I'll have to do it.
I had the idea that you could also try running a Linux guest but saw you've done that but with Windows guest. Quote:
You also insisted on using pci.1 instead of pci.0. Indeed, I told you that with VFIO, the pci.1 GPU did not work for me... Pegasos loads the Kickstart and then gets stuck forever.
I wonder why. Are there any logs from kernel/driver using the debug kernel and setting -append 'serial debuglevel=12' or similar? (I never remember which is a good debuglevel. Setting it too high will crash and won't boot, too low will not log enough but can't remember what is the still acceptable highest level.)
Sadly, the Picasso96 API doesn't allow the driver to do the same for the graphics.library. ... The driver falls back to CPU-based copies to/from VRAM for all other motherboards.
So it looks like we should try to optimise this case as it seems to be used frequently.
Quote:
it has copy routines that are optimmized on a per-CPU basis. So, it'll use altivec based copy routines on CPUs with altivec, and 64-bit float based routines elsewhere. It'll even use the A1222's SPE.
Do you know what AltiVec ops are used? I've seen vperm in one profile which I think also can swap bytes around that may be used for endian conversion. I don't know how well it's emulated but this sounds like it could fall back to byte access in some cases. If this has any effect could be tested by using -cpu 750cxe where then FPU regs would be used (which might also be slow but as long as no FPU ops are done maybe not) but then we would need benchmark results with both -cpu 750cxe and without -cpu option that defaults to 7457 but I haven't seen those yet. There are some patches on the list that change these AltiVec instructions (although only the way they are implemented not the results) so I'll wait until they are merged then can try to find where it's implemented and what it does. (But maybe those instructions that are used by AmigaOS are already converted, I don't know.) If you want to look at it it's somewhere in qemu/target/ppc probably in a vsx.impl file. QEMU has some support for using 128 but quantities that could be translated to host code but it's possible the VSX/AltiVec instructions don't use them and could be optimised.
Quote:
I already have: it's called GfxBench2d. It includes some of the same copy routines used by the graphics.library and RadeonHD/RX. It benchmarks multiple copy algorithms, although they're NOT all uploaded to the HDRLab website. Check the text output.
OK, then can you make a small C program that just measures these copy routines that can be compiled on Linux? QEMU has two modes, one is full system emulation with qemu-system-ppc but it also has user mode with qemu-ppc that can run on Linux or BSD executables compiled for different architecture. With that we could easily check the compiled code, because we won't have to find it between all the other OS code. So if you have a test of just these copy routines that can be compiled for PPC Linux you could use -d options with qemu-ppc running on x86_64 Linux to get the guest and host asm and compare them to see how these are transformed. Finding it in the qemu-system-ppc output is probably impossible so we'd need a seprate small test case for that.
Quote:
- What's the VFIO performance when running Linux (PowerPC) inside QEMU? Do they also have a big performance hit?
I've asked to test this, haven't seen an answer yet but we got Windows results now. Getting the same tests that @Georg did or getting AmigaOS results from @Georg so we have both host and guest results from same hardware might help though so we don't have to compare numbers from different machines. Quote:
- What does the generated host code look like? If it isn't a tight loop, then the CPU/PCIe-controller won't be able to batch up transfers into larger packets
See above, I think we would need a small Linux test executable for that then you can get generated code from qemu-ppc. Or you could try to find the implementation of the VSX/AltiVec ops and work out how they are translated but that may be harder than making a test case. Quote:
- How does VFIO even work? Can the generated host code access the hardware directly? Or is there some kind of abstraction layer in-between. If there is a layer in-between, what's its overhead? For example, do accesses run through an exception handler?
We haven't seen guest exceptions (I've asked for info irq results which we got but showed no excessive guest exceptions), I don't know about host exceptions. I don't know how vfio works, you could ask on QEMU list or find some docs but I think what it does is set up the IOMMU on the host to map the card into the guest's address space so guest code would access the card directly. This may depend on host IOMMU and may fall back to some slower ways when that's not available or not work at all. So theoretiaclly it could also depend on host and host firmware (BIOS/UEFI).
@nikitas When testing with x86 guests it might use KVM so guest code runs on the CPU which might give different results. You can add -accel tcg to force it to translate x86 code the same way as it does PPC code which will be slower but might be closer to the PPC guest case. Although PPC and x86 ops are different so the translation is also different but at least it goes through more the same process as with PPC guest.
I've always wanted to batch the draw ops, but see no safe way of doing so with the graphics.library, because it has no Flush() or Finish() function to flush the queue (i.e., apps/games expect draw ops to happen immediately, and never signal a flush/finish).
This P96 *.chip or *.card function should be used by IGraphics->WaitBlit().
It's not just flush/finish but additionally waiting for completion, but since it has to be used between any GPU rendering and CPU rendering, without using it the CPU based rendering functions may work on wrong data, it may help if several GPU functions are called in a row.
Quote:
A-EON owns the driver, so the source code is on their repository. Multiple people have access to it, although it's rare for anyone other than me to touch the code.
I may have access to it, not sure, but I never checked your driver sources. Knowing how the P96 API, incl. the OS4 extensions, works is enough to know what a gfx card driver can do and what it can't.
IIRC only the Vector Load and Store (Indexed) instructions.
Additionally the Vector Data Stream Touch instructions for pre-fetching data (read) and allocating cache lines (write) (similar to DCBA on CPUs without AltiVec, or on CPUs without DCBA support DCBZ as replacement), but those may only help on cached DRAM, not on cache-inhibited VRAM.
Even without using the streaming instructions using 2 vector store instructions in a row will skip the usual "read 32 byte cache line"/"modify cache line"/"copy back cache line" used on PPC and just writes the 32 bytes. For anything smaller, for example 4 64 bit double or 8 32 bit integer stores in a row in a memcpy() loop that's not the case and it's using the slower read/modify/copy back cache line method. May only help on cached DRAM, not on cache-inhibited VRAM, as well.
News feed
OS4 Feedback forum
OS4Depot Feedback forum
Amigabounty forum
OpenAmiga forum
AmiCygnix forum
ABC shell forum
AmiKit forum
Cinnamon Writer forum
CodeBench forum
Digital Universe forum
Dopus 5 forum
E-UAE forum
Gnash forum
Ibrowse forum
JAmiga forum
Odyssey forum
OWB forum
Qt forum
SmartFileSystem forum
Timberwolf forum
TouchDevice forum
TuneNet forum
Unsatisfactory Software forum
Donate
