QEMU GPU vfio-pci pass through

Returned R9 270 GPU had problems.
The seller was from my city and simply returned my money.
The curse of AmigaOS does not want me to do this emulation

Anyway I will look for another one.

I followed this guide the steps worked but the GPU had problems. Even with Windows 10 I do not know why but it corrupted the Windows 10 filesystem.

I like to share so if someone can see it and wants to give their contribution with the steps it is ok

Even if there are many guides around.
So as to have at least a basic guide.

I also leave the link where I was seeing the next steps for (vfio)
Also @balaton @joerg and others can take a look and see if it is useful.

https://forum.level1techs.com/t/ubuntu ... o-vfio-quick-start/212639

Thanks (I'll get back to this topic at post 100 as soon as I find another R9 270X GPU (I hope).

@white
If you just want to experiment and only want 2D maybe you can also try a simpler card before spending more on it. For that, any card that has an AmigaOS driver could work. I've found locally a HD6450 for 2.50 Euros that should work as good as any other. (It has PCI ID 1002:6779 and subvendor shows it was in some HP machine originally, these seem to be available even at half that price; it is a slow card but at least does not draw much power so for experimenting should do. I could get RX cards for ten times that much around 25 Euro but they need additional power and I have no driver for those so why bother.)

I have this HD6450 card but it has a noisy fan so I removed it from the machine after trying a bit so I can't test it now. I could not fully make it work because I've found all kinds of problems but most of those could be solved. First it froze the host when the BIOS emulator tried to run the ROM of the passed thorugh card or messing up the host display. This was that due to the Intel iGPU not playing well with VGA ports (look for vgaarb messages in syslog to diagnose this). This was seen by somebody else before here and avoided by not using x-vga=on but that's not correct as then the BIOS ROM cannot fully init the passed through card. This could be solved with a kernel patch that fixes vgaarb with i915 driver that I've found before and linked in some other thread or maybe it works better with newer kernels but I did not test that. I also had an even older ATI (not even AMD yet) X550 card that I've added instead of the iGPU which then did not have this problem but it's not something I want to keep using so this was just for testing. (If you don't use iGPU but have two gfx cards you probably won't get this.)

Then found that sam460ex did not work due to its firmware expecting RadeonHD cards on PCIe port and getting confused when seeing one on the PCI bus (this is a bug in the firmware IMO as it looks at PCI port instead of device ID to detect RadeonHD but that works for physical cards as you can't plug one into PCI unless you're on a Sam440 maybe and trying to use a bridge; maybe it should look for device IDs instead). It might work better in the future when I fix the PCIe ports on sam460ex emulation but I did not bother with that yet. Then I tried pegasos2 which had some issue with memory regions clashing or something like that so I think it could not map the card. Maybe it would work with using bus=pci.0 but that had some IRQ problems at the time but I think I've fixed that after that so may work now. Finally I tried amigaone which had the most success as it saw the card and the firmware displayed on it and started to boot but then it turned out the AmigaOne version of AmigaOS lacks the RadeonHD driver on the install CD so it did not fully boot so I don't know how well AmigaOS would have run. I gave up at that because I did not want to spend more time on doing an install and adding a driver and could not use BBoot as this needs the firmware to run to init the card and BBoot did not run under AmigaOne firmware at the time.

So it's not easy to set up but it's possible. I may get back to this at one point but I don't want to add two cards permanently and I need to repair the fan or get a passive cooled card (the X550 is passive heatsink only but has no AmigaOS or MorphOS driver and did not work before with QEMU either but there are passive HD cards too). I wanted to write about the above in a more useful way but had not time so now you have it as a brain dump to at least give some hints where to look for different errors.

In the meantime there were some patches in QEMU for vfio so try with the latest rc version (currently 10.0.0rc0), these were mainly for Intel iGPU but I think it may have also fixed older ATI/AMD cards such as X1300 and older like 9550 or so which may work better with AmigaOS but those are mostly AGP so it might be difficult to find one that you can connect but there were some PCI or PCIe ones so those could also be tested again. The one X550 I have from that era does not seem to be supported by AmigaOS nor MorphOS, although it's very close to X700 which may be supported.

The list of AMD GPUs page on wikipedia is useful as it tells you which card has which chip, otherwise it's hard to identify the marketing names such as R9 270 that has nothing to do with the chip (Pitcairn XT) and the family (GCN 1st gen) which is what the drivers refer to. So you need it to find out if a card would be supported by the driver and which driver. E.g. HD8870 and R9 270 may actually be the same card just renamed because NVidia changed names as well or something like that. The HD6450 may be the same as HD7450 or HD8450 and RX 460 may be the same as RX 560.

When searching for vfio passthrough setup guides avoid anything that talks about virsh, virt-manager or has XML fragments. These just complicate things. Look for something simple such as:
GPU-Passthrough with Qemu (without libvirt)

I hope that's more than enough info in this post to avoid more questions

@balaton

Thanks for the reply and suggestions,
Yes consumption is important.

But I have to put the card on the same PC where I have the 4060 which consumes about 120W.

The power supply I use is a Corsair 750RMx Gold 80 Corsair Modular
Among the cables I have in the box still to use there is this:







I bought a Sapphire AMD Radeon R9 280x Dual-X 3Gb but it requires 8+6 PIN so it seems 250W. (price 30 euros) he told me it works.





Obviously it is the last card I buy this is exactly the one in the list of:

https://hdrlab.org.nz/projects/amiga-o ... r-hardware-compatibility/

If it doesn't work, patience I will dedicate myself to something else

@All @Hans

It would be nice to make a driver just to have 2D acceleration with Composite.

With an old Radeon card from the ones listed.

And then use 4.1 as it was once done with the Motorola 68.xx with the Workbench without PicassoIV but only with AGA.

Without writing any line of code that enables 3D
I wonder if it would be easy to do it and maybe easier to adapt to emulation.

I don't want to offend anyone who rightly uses 3D games on AmigaOS 4.1 and other programs after buying a real machine.

Question
But would a driver with these features be possible and easy to make ?

@white
Quote:Not possible. While compositing is a 2D gfx API of graphics.library it's using the 3D features of the GPU for it.
The only GPUs supporting compositing on AmigaOS are R100/R200 based Radeon gfx cards (very old, PCI and AGP only, driver ATIRadeon.chip, for example Radeon 9250), Radeon HD gfx cards (RadeonHD.chip driver, for example R9 270X) and Radeon RX gfx cards (RadeonRX.chip, for example RX 580).

Of course there is fallback code in graphics.library using CPU based software rendering for compositing instead on unsupported GPUs, but unless you have a QEmu system on which the emulated PPC CPU speed is at least 10-20 times faster than the CPU speed of a X1000 or X5000 it's probably not realy usable, at least compared to a system using the very much faster GPU 3D units for it...

@joerg
Thanks for the very clear and detailed explanation.
At least I give up.
Now I'm looking at the power supply with the 8+6 cable I haven't seen yet.
When the card arrives I'll look

One is definitely busy the other is free

I spoke with Corsair technical support they told me there shouldn't be any problems.
I explained to him that I had to put an rtx 4060 and a Sapphire AMD Radeon R9 280x Dual-X 3Gb but it requires 8+6 PIN
Obviously I have to use two separate cables.





I guess these are the PCIe sockets
I knew I shouldn't have gotten 8+6 to avoid complicating my life

Ok done I had to disassemble the PC
I added a PCIe cable to test I connected the 4060 and 2060 together with two separate PCIe cables
So in theory it should work fine for the R9 280x Dual-X
(let's hope).

Sapphire AMD Radeon R9 280x Dual-X seems to be compatible with driver 2.10

(Sam460ex) it's just a note for Sam460ex emulation

AMD Radeon Sapphire Dual-X R9 280X 3GB GDDR5 OC Tahiti XT2/XTL Driver 2.10 Working in HDRLab


I was looking through my AmigaOS stuff,
and I also found these drivers:

RadeonHD.chip Driver Version 2.10

Without going into the entire Readme

REQUIREMENTS
-----------
- A supported graphics card:
- Radeon X1300-X1950 graphics (not recommended)
- Radeon HD 2000-7000 series graphics card (excluding the Radeon HD 7790)
- Radeon R5 230/235/235X
- Radeon R7 250X/265
- Radeon R9 270/270X/280/280X
- AmigaOS 4.1
- An AmigaOne-X1000 (with the latest firmware), or an AmigaOne-500 (or Sam460ex)


INSTALLATION
------------
Simply run the included installer.

And then also included this:

W3D_SI.library Driver Version 1.14

ABOUT
-----

The W3D_SI.library Warp3D driver provides 3D graphics acceleration for selected
Radeon HD 7xxx and Radeon Rx series graphics cards. It supports cards that you have
Southern Islands Graphics Processing Units (GPUs). A list of supported cards can
be found below.


REQUIREMENTS
------------

- A supported graphics card:
- Radeon HD 7750-7970 series graphics card (excluding the Radeon HD 7790)
- Radeon R7 250X/265
- Radeon R9 270/270X/280/280X
- IMPORTANT: Please use a graphics card with GDDR RAM; cards with DDR3 RAM
have been reported as not working on AmigaOS
- AmigaOS 4.1
- An AmigaOne or AmigaOS compatible computer

NOTE: MiniGL 2.20 or higher is recommended


INSTALLATION
------------

Simply run the included installer.

@balaton @joerg @all


The GPU should arrive tomorrow because the person who sold it to me is practically 3 hours from my city.

Many years ago I also used BackBox, an Italian Linux distribution.

However, I think it is appropriate to sign up to some Linux Forum in order to work more easily.
And maybe with some support.
Many years ago there was a Linux User Group in my city.
But now I am 56 years old

In any case, there is the pleasure of trying to do something new with vfio.
And spend some time, let's say it's like the game Elden Ring
Between one curse and another, little by little I will try to do something.

I am a spontaneous person, I could consider paying a Linux expert from my city, to get the "vfio gpu passthrough" directly on my computer and then do the Backup once the work is done.

It could be a solution.
On the other hand, why not
It's like buying anything else.

On the other hand, just like you buy drivers, players and other things, you can buy a configured GPU in vfio.

The cost should also be low if someone knows what to do.
At least to have a base.

And then move to the AmigaOS side later once I get the vfio gpu

I don't think it's a bad idea for me who is 56 years old and has many pending things to do.

Obviously the question is for informational purposes
regardless of the AmigaOS emulation.
The main goal is to isolate the IOMMU GPU make it available with VFio and then as a second phase is the one dedicated to AmigaOS.
Is this the concept ?
If I understood correctly ?
Thanks.

Recommended movie if you don't know what to watch:

The Electrical Life of Louis Wain

English Trailer
https://www.youtube.com/watch?v=xzDr_tbL-es

I just bought MGS vol.1 for PS5 and yet I have emulated them all.
But I bought them perhaps in the hope of going back to being younger (but time passes)

And I miss my wife so much but I'm happy because she doesn't suffer anymore.
It was a sweet journey with a soulmate who is the rarest thing to find in your life. (at least for me).

@white

This is a picture of a 6+2 PCI-e cable. They are designed to be either 6 or 8 pins depending on what your GFX card needs. You can see there are 2 pins separate from the 6 pin block, which become a 8 pin connector when put together.

You will need 2 of these cables for the R9 280X. The first cable will use all 8 pins when plugged into the GFX card, and the second cable will only use 6 pins.

This is all in addition to the cables required for the other GFX card.

@derfs

Thank you for the valuable suggestion
That was exactly what I was worried about
But fortunately the single cable is perfect for 8+6
Here it is

the rtx 4060 zotac OC only requires 8 PINS while I got the Ryzen 5800x when the 3060Ti couldn't be found and I kept the 1060 6gb which was perfect for 1080p.

If I had a 3060Ti now I wouldn't have been able to use the second cable
for another card










it's more precise that way

While the rtx 4060 is connected with another cable

You were very precise, thanks

The 2 PCie cables are the original ones included with the Corsair power supply.
Not purchased on Amazon or Ebay etc. apart

Obviously until I connect there is a bit of fear but I should be fine.

Too bad, today is Saturday
So the delivery will be on Monday

I leave some links that may be useful for the purpose and if anyone has some links to provide me as a guide IOMMU GPU VFIO etc. for qemu, I like to read.

https://www.theseus-os.com/Theseus/boo ... hine/pci_passthrough.html

https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

https://terenceli.github.io/%E6%8A%80% ... 19/08/31/vfio-passthrough

https://terenceli.github.io/archive.html

https://github.com/benbaker76/linux-acs-override

https://mathiashueber.com/passthrough-windows-11-vm-ubuntu-22-04/

Thanks.

Note:
In the bios Asus TUF GAMING PLUS B550 PLUS
there is the ACS item (it must be set to ENABLE)



To have separate IOMMU groups, your processor needs to have support for a feature called ACS (Access Control Services). Make sure you enable the corresponding setting in your BIOS for this.

If you don't have dedicated IOMMU groups, you can try moving the card to another PCI slot.

Should that not work, you can try using Alex Williamson's ACS override patch. However, this should be seen as a last option and is not without risks.

As of writing, the ACS patch is part of the Proxmox VE kernel and can be invoked via Editing the kernel command line.

ACS allows PCIe devices to always pass through the IOMMU, providing isolation and thus better granularity of IOMMU groups.

A curse looms over emulation
I've been waiting for the GPU for 5 days.
And yet the seller is 2 hours from my house.
Tracking the package now the package ended up far away 5 hours from my house stuck in a HUB.

In the meantime I prepared the system and read a few things.
In the last compilation of "Beta 10" I had this error:

sing './build' as the directory for build output
python determined to be '/usr/bin/python3'
python version: Python 3.10.12
mkvenv: Creating non-isolated virtual environment at 'pyvenv'

*** Ouch! ***

found no usable tomli, please install it

Solved like this from shell:
pip install tomli


This is the current configuration:

ZOTAC GAMING GeForce RTX 4060 8GB Twin Edge OC
Sapphire Dual-X R9 280X OC
Ryzen 5800x
Corsair Hydro Series H80i v2 liquid cooling 120mm
ASUS TUF GAMING PLUS B550
PSU Corsair 750RMx
Samsung SSD 500gb only for linux
Corsair Vengeance Ram DDR4 2x 16gb 3600Mz PRO RGB
Corsair Vengeance Ram DDR4 2x 8gb 3600Mz LPX

Corsair is a smart guy in the PRO RGB version they put the "Micron Technology" Rams that can be overclocked up to 3800Mz without any effort.

While in the LPX version Corsair put the "Samsung" ones less prone to overclocking.

But the CAS latencies are identical so since I'm not a fan of overclocking
once I've done the tests I'm using them together.

So 48gb of Ram in all this will help me to create a parallel virtual machine and do the tests on the fly to see the results.

32gb for Linux
16gb for Parallel Virtual Machine

if the GPU arrives

Of course @Hans probably already knows if the GPU will work or not.
But I'm interested in having fast Composite 2D.
Worst case scenario I'll use the GPU as a knick-knack

@Balaton
Thanks I hadn't seen the link you left me for GPU vfio

https://kolahilft.de/2021/11/gpu-passt ... ith-qemu-without-libvirt/

It finally arrived,
I'll see if everything is ok tonight.

The model is this:

Sapphire Radeon R9 280X 3G GDDR5 OC Version (11221-00)
Tahiti XT2/XTL
Driver 1.2

On hdrlab it says 11221-00-25G
On mine as you can see in the photo it says 11221-00

on:

https://hdrlab.org.nz/projects/amiga-o ... r-hardware-compatibility/









Let's hope it works

@white
At least the cat seems to like it.

@balaton
Thanks
Yes the cat was happy
Worst case scenario I'll let her play with the fans if I can't do anything with it

Apart from that I paid 30 euros + shipping for the card.
I wanted to try PayPal "pay in 3 times" for the first time
So PayPal accepted my request
And in reality it's 3 payments of 12 euros.
Obviously not for the 30 euros I wanted to try this method maybe for more serious purchases in the future.
Currently my PC is fine like this.

I also have Windows 11 IOT-LTSC 24H2 installed and activated and backed up (but I don't like it). and I don't use it for the moment.
If I really have to change it, it's already activated.

Honestly I prefer Windows 10 LTSC.

I find the most difficult thing to do now is to understand "boot.fth"
Because I've never used it.
I need to understand how to configure it while "vfio" is documented

I still don't understand much about how to use boot.fth.
I didn't follow the forum when it came out.

Well last Backup and then we start trying

Amd Gpu is recognized without problems by Windows 10.
Radeon Win 10 Win 11 64Bit Legacy 22.6.1.0 drivers
The other card that I had previously used and then returned had problems with the BIOS and also did not complete the installation of the AMD drivers.
This one was recognized without problems and does not create compatibility problems in Windows 10 with rtx 4060.
On Windows I can also disable it.
It was to verify the correct functioning.

I will return to post number #100 of this Topic so as to keep the information and the various settings on qemu organized.
So that the topic does not become dispersive.

It is useless to tell people who are experts or who are trying something similar that any suggestion is welcome.
I like to share information because it is important.

well obviously not those of the credit card

The cables for connecting the PINs worked without problems.
Now I have the doubt of whether to force the Radeon to PCI 8x in the BIOS
Why does the rtx 4060 work by default only at PCI 8x

And just to avoid incompatibilities?
Currently the BAR is disabled

Also now I took a look at these settings in the BIOS and the related documentation.
ACS ENABLE
PCIe ARI SUPPORT
PCIe ARI ENUMERATION
PCIe TEN BIT TAG SUPPORT

@Balaton
@Joerg
@all

I only had time yesterday to try a bit
Why doesn't it tell me:
Vfio Kernel in use

But I haven't installed qemu yet because I was thinking about the Radeon GPU

does it depend on that or do I have to do some additional steps to associate the Vfio with the kernel ?

Or when I install qemu does it associate it with the kernel ?






I did something quick, I compiled the BETA 10 of qemu.
Then I calmly start to see if there is any possibility.
I only took a small step

I did something quick, I compiled the BETA 10 of qemu.
Then I calmly start to see if there is any possibility.
I only took a small step
A small starting point.

Now I'm going to watch "The 3-Body Problem" taken from the book trilogy, too bad they don't make these series complete right away.

Highly recommended "After Life" a deep and sad TV series that helps you reflect.
I recommend it, there's nothing wrong with it.

BIOS IOMMU Enabled
ACS Enabled
BAR Disabled ( enabling it makes the bios go into a continuous reset loop if i mounted Nvidia and Radeon together )

Verified that IOMMU was working
sudo dmesg | grep -i -e DMAR -e IOMMU

sudo dmesg | grep -i -e DMAR -e IOMMU
[sudo] password di white:
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.14.0-13-generic root=UUID=946340b4-4e55-43ee-a318-732651a7c014 ro amd_iommu=on iommu=pt vfio-pci.ids=1002:6798,1002:aaa0 quiet splash crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M vt.handoff=7
[ 0.035931] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.14.0-13-generic root=UUID=946340b4-4e55-43ee-a318-732651a7c014 ro amd_iommu=on iommu=pt vfio-pci.ids=1002:6798,1002:aaa0 quiet splash crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M vt.handoff=7
[ 0.670261] iommu: Default domain type: Passthrough (set via kernel command line)
[ 0.703176] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.703232] pci 0000:00:00.0: Adding to iommu group 0
[ 0.703254] pci 0000:00:01.0: Adding to iommu group 1
[ 0.703272] pci 0000:00:01.2: Adding to iommu group 2
[ 0.703293] pci 0000:00:02.0: Adding to iommu group 3
[ 0.703314] pci 0000:00:03.0: Adding to iommu group 4
[ 0.703329] pci 0000:00:03.1: Adding to iommu group 5
[ 0.703350] pci 0000:00:04.0: Adding to iommu group 6
[ 0.703374] pci 0000:00:05.0: Adding to iommu group 7
[ 0.703395] pci 0000:00:07.0: Adding to iommu group 8
[ 0.703411] pci 0000:00:07.1: Adding to iommu group 9
[ 0.703432] pci 0000:00:08.0: Adding to iommu group 10
[ 0.703447] pci 0000:00:08.1: Adding to iommu group 11
[ 0.703474] pci 0000:00:14.0: Adding to iommu group 12
[ 0.703488] pci 0000:00:14.3: Adding to iommu group 12
[ 0.703557] pci 0000:00:18.0: Adding to iommu group 13
[ 0.703572] pci 0000:00:18.1: Adding to iommu group 13
[ 0.703590] pci 0000:00:18.2: Adding to iommu group 13
[ 0.703604] pci 0000:00:18.3: Adding to iommu group 13
[ 0.703619] pci 0000:00:18.4: Adding to iommu group 13
[ 0.703633] pci 0000:00:18.5: Adding to iommu group 13
[ 0.703648] pci 0000:00:18.6: Adding to iommu group 13
[ 0.703662] pci 0000:00:18.7: Adding to iommu group 13
[ 0.703698] pci 0000:01:00.0: Adding to iommu group 14
[ 0.703714] pci 0000:01:00.1: Adding to iommu group 14
[ 0.703731] pci 0000:01:00.2: Adding to iommu group 14
[ 0.703736] pci 0000:02:00.0: Adding to iommu group 14
[ 0.703741] pci 0000:02:08.0: Adding to iommu group 14
[ 0.703747] pci 0000:02:09.0: Adding to iommu group 14
[ 0.703757] pci 0000:03:00.0: Adding to iommu group 14
[ 0.703762] pci 0000:03:00.1: Adding to iommu group 14
[ 0.703766] pci 0000:05:00.0: Adding to iommu group 14
[ 0.703794] pci 0000:06:00.0: Adding to iommu group 15
[ 0.703812] pci 0000:06:00.1: Adding to iommu group 15
[ 0.703828] pci 0000:07:00.0: Adding to iommu group 16
[ 0.703845] pci 0000:08:00.0: Adding to iommu group 17
[ 0.703861] pci 0000:08:00.1: Adding to iommu group 18
[ 0.703877] pci 0000:08:00.3: Adding to iommu group 19
[ 0.703893] pci 0000:08:00.4: Adding to iommu group 20
[ 1.704259] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002f8900]
[ 1.704436] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).


qemu-system-ppc: -device vfio-pci,host=03:00.0,bus=pci.0,x-vga=on: vfio 0000:03:00.0: failed to open /dev/vfio/14: No such file or directory

I can't figure out what file I need to configure to fix this


Verified that IOMMU was working
sudo dmesg | grep -i -e DMAR -e IOMMU

sudo dmesg | grep -i -e DMAR -e IOMMU
[sudo] password di white:
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.14.0-13-generic root=UUID=946340b4-4e55-43ee-a318-732651a7c014 ro amd_iommu=on iommu=pt vfio-pci.ids=1002:6798,1002:aaa0 quiet splash crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M vt.handoff=7
[ 0.035931] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.14.0-13-generic root=UUID=946340b4-4e55-43ee-a318-732651a7c014 ro amd_iommu=on iommu=pt vfio-pci.ids=1002:6798,1002:aaa0 quiet splash crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M vt.handoff=7
[ 0.670261] iommu: Default domain type: Passthrough (set via kernel command line)
[ 0.703176] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported

is there any error in the .log list above ?

a little help ?