PCI device memory from BAR under qemu alwys zero

Does anyone know how to read out the memory of an PIC device defined via a BAR?

I find my PCI device and call on that the GetResourceRange, and dumps the memory like this:



But regardless which PCI device and matching bar I choose. I always only read zeros.
And at least for some device I know there must be some values presents.

What am I missing?

@MigthyMax

Use resourceRange->BaseAddress to obtain the BAR base address.

And then add the offset to your registers of interest.

@geennaam

Even using BaseAddress doesn't result in other values but zeros. But I think my problem is that i running under qemu.

Dumping memory on a real X5020 dumps even values other than zero.

BTW in which case you have to use the physical instead of the base address? (Even that they are most of time the same?)

@MigthyMax

Unfortunately the autodocs don't contain information about the difference between Physical and BaseAddress.

I can imagine that "Physical" is the address in PCI space and BaseAddress is the address in CPU space. They can be the same but this is not always the case.
It could also be that Physical is meant for Memory space BARs only because they have to be mapped in a physical memory range where IO BARs can be mapped beyond physical memory range and can have any address.

Anyways, someone should update the autodocs

Why it isn't working in QEMU is probably because you are trying to access a device on a bus that is not properly emulated.

@geennaam

I hope that the virtio-xx-pci devices are correctly emulated

At least all vendor specific capabilities etc I find, and can read. These have the following structure:



And all values I retrieve for them looks correct (and even don't consist only out of zeros)



Just that the bar memory is zeroed on qemu.

@MigthyMax

You're reading the extended capabilities within configuration space. You're not in BAR space.

If the virtio device is handled correctly by the target firmware then reading BARs should work.

At least for VFIO you can access the BARs of physical PCIe cards. But there was an issue with 64bit bars for a Pegasos2 target. This requires some fiddling with scripts or the altenative firmware called BBOOT. The SAM460 target doesn't emulate the PCI buses correctly.
But I do not know how the virtio devices are handled in that target.

@MigthyMax
Quote:Probably, but did you configure it manually for example with the U-Boot "pci write.b" commands?
I doubt U-Boot includes support for QEmu virtio devices, but you can check if U-Boot did configure something with
and with the found virtio device for example
(if it's bus 0, device 1, function 2)

@MigthyMax

Are you emulating a Pegasos II? If so, be aware that the Pegasos II kernel couldn't handle 64-bit BARs. So, the memory BAR would be missing for virtio devices. I say "couldn't" because I have fixed the kernel PCI scanning code.

In the meantime, there are workarounds to get it to work which basically involves changing the OpenFirmware device properties to make the 64-bit BAR look like a 32-bit one. I have a boot script that does this (which I don't need any more). No idea if BBoot does the patching automatically.

Quote:
Yes, they are.

Hans

@geennaam

Quote:
I guess it's assumed that, if you're writing driver code, then you know what virtual and physical memory are. The BaseAddress is the virtual address (i.e., the address you use in code running on the CPU). The physical address is where the PCI card is located in physical address space.

It may be a little confusing that the physical address is 32-bit, when the actual physical address space is 64-bit. This is for legacy reasons, and is also rather limiting.

Hans

@Hans

The issue with the SDK in general is that the authors assume that the audience is familiar with the complete history of RKRMs or Amiga programming in general. And that the autodocs and wiki are a guide to what's changed since the 68k era.

In this case, it's the names of the fields in combination with the contents of the autodoc that are confusing.

From the autodoc:


The "Physical" field is only mentioned in combination with MEMORY SPACE ACCESS.

It tells me that I have to take the "physical address range" into account by obtaining the "Physical" field. It was my assumption that it tells me something about the actualy physical size of the BAR instead of allocated size (Size field). Some cards allow to use the unallocated/unused BAR memory as user defined memory. So up to now, I've ignored this value and always used the BaseAddress. For my drivers, there's no need for the physical address of a BAR anyways. Because PCIe cards come with their own DMA engine which you have to feed with the physical address of a buffer in main memory. Card to card transfers are in general not used/possible with PCIe.

Maybe the physical address is needed for a gfx card if you want to use the host DMA engine to transfer data from host memory to GFX memory. And if it was, I would have obtained it from the MMU in supervisor mode. Just like i do to obtain the physical memory address of a buffer in main memory.

But if you look at this structure in pci.h, the comment makes it clear that it's not a range but a "Physical base address of the range"

@all

A lot of information, which I currently cannot sort in.

@Hans @joerg

I'm using the Pegasos 2 emulation with bboot. I'm added the virtio devices as following to the working qemu call:



I had the thought that qemu knows that "-m pegasos2" is 32bit, thus the virto devices will be 32bit too?

So what do I have to tinker to get them mapped correctly? What kind of boot script?

@MigthyMax

The instructions are buried somewhere deep in this thread. You need to tailor it to your hardware. I'll see if I can dig up the script I used.

Hans

@MigthyMax
BBoot should patch 64bit BARs to change their type to 32bit so the pegasos2 AmigaOS kernel recognises it. It should also tell you what it did, just read the debug output:

The meaning of the numbers are explained in the BBoot README and OpenFirmware PCI binding docs. If you use pegasos2.rom then you can still use BBoot with bboot.fth to apply the BAR patches or you can do the same with a Forth script instead. (If you use BBoot with a newer kernel and don't want it to touch the PCI config then you can use "Os V5 Ab" bboot opts string instead of the default "Os V5 Apb" (Apb is Action PCI, Boot so removing p from it will skip that and only boot without patching or printing PCI data)).

You should probably check with Ranger how the AmigaOS kernel sees the BARs of the device and can use info pci and info qtree commands from the QEMU Monitor to see what's their status on the QEMU side.

@balaton

I read multiple times what you wrote and even found bit and pieces in the source of bboot.
But I still no clue who and where to adopt the bar truncation in my setup with bboot and
initrd Kickstart.zip file. Any example anywhere (expect as Hans mention buried deep done
in the mention thread)

@MigthyMax
With BBoot you don't need to change 64 bit BARs to 32 bit yourself, BBoot does that.
Check what Ranger on AmigaOS displays for the virtio device you are trying to use, if Ranger displays it correctly, incl. the memory BAR(s), something is wrong in your code.

@geennaam
Quote:The AmigaOS expansion.library PCI functions were created for PCI cards only, and apparently never updated for PCIe support/differences.

I checked the address which i retrieved with the address from Ranger etc. This seems to be correct. And even running my test on a real machine, gives me even. nore zero values. I'm not sure what I'm doing wrong.

It might even be that on qemu the virtio device isn't yet correctly configure and this not data present. There might be issue with the synchronization between cpu and memory of the device. There are so many pitfalls.

Is there anywhere an simple example how to drive a device device? especially reading from the BAR memory? Or a tool known to work, which dumps BAR memory?

@MigthyMax
Quote:AFAIK when using BBoot it doesn't execute the Pegasos2 OpenFirmware.

AmigaOS does not configure any PCI cards(*), it relies on the firmware (U-Boot, CFE, Pegasos2 OpenFirmware, etc.) doing it correctly and uses whatever it gets from the firmware.

A few AmigaOS PCI drivers, for example the Voodoo 3/4/5 gfx card driver, can configure their PCI cards themselves even if the x86 BIOS ROM of the PCI card wasn't executed, but most can't.
Maybe that's the case for all AmigaOS PCI drivers used in QEmu until now (SM502, RTL8029/8139/8169, a1ide/peg2ide), but for example if someone implements Radeon R100/R200 support in QEmu the AmigaOS ATIRadeon driver can't work if the BIOS wasn't executed by a firmware x86 emulator.

I don't know anything about QEmu virtio PCI device emulation, but either it has to include a x86 BIOS ROM for configuring it, in which case you can't use BBoot but have to use the Pegasos2 OpenFirmware instead and fix the 64 bit BARs with a forth script, or if they don't include a BIOS you have to configure it in the AmigaOS driver you want to implement for it.


*) The only exception is the classic Amiga version of AmigaOS 4.x. Sine there is no firmware it includes a x86 emulator executing the BIOS ROMs of the PCI cards. Additionally it's a Zorro III<->PCI bridge (Mediator, Prometheus or GRex) on classic Amigas and there are differences to systems with PCI support.

@MigthyMax
Quote:

I don't know much about virtio devices but it's likely the driver needs to configure the virt queues and let the device know about it before it would receive any data from the device. Here are some links that may have some useful info:

https://www.redhat.com/en/blog/deep-di ... -networking-and-vhost-net
https://www.redhat.com/en/blog/virtque ... tio-ring-how-data-travels
https://marz.utk.edu/my-courses/cosc562/virtio/

and the ultimative documentation is the virtio spec but that's a bit long and may be overwhelming at first but that should be your definitive source of information.

@joerg
Quote:

When using BBoot with -kernel (i.e. without pegasos2.rom so running under QEMU VOF as BBoot could also be used with pegasos2.rom) BBoot will assign addresses to PCI BARs and program the BAR registers accordingly, as seem in the debug output it prints. This is because AmigaOS kernel does not do this unlike Linux or MorphOS which scan PCI devices and assign addresses on their own. So even when booting with BBoot PCI devices should be set up and show up in Ranger. The only part BBoot does not do is to run the BIOS option ROM of the card but this should not be needed for virtio devices. The virtio-net-pci device may have a ROM but that's only to support network booting on pc machine so likley would not work with pegasos2.rom anyway and the device itself should not need any init done by the ROM. The driver should detect and then init the device according to the virtio spec on how virtio-net should work.

Quote:

I'd say the need to run the ROM is the exception for some gfx cards rather than the norm and the reason for that is that these GPU devices might require quite complex and undocumented init procedures that thier ROM implements so that's the easiest way to get them working but this is not normally required for other devices, especially virtual ones that usually don't need any init from the guest other then the driver establishing the communication channels in the shared memory. As for emulated R100 I think that should work even without running any ROM because what the ROM does is setting up card clocks and memory controller (basically booting the hardware on the card) and set up screen timings all of which make no sense in an emulated envirionment where these aren't needed. So this can just be skipped and the driver can start from there and the device can be emulated in the state where ROM would have left it on real machine so this won't be an issue even for ati-vga. The issue for ati-vga now is just not emulating enough of it yet for the driver to be able to use it.

Quote:

I don't know for sure either but I think even if it has BIOS ROM that's only for network boot and should work without that (after all virtio-net is used on all other emulated machines where no BIOS ROMs are used). So no need to use pegasos2.rom or do anything with the BIOS, the driver just needs to set up the device and start communicating via virtqueue. This may need writing some stucts in the device memory (you'd use the virtual base address to access that) then let the device know where it finds the queue (which may need the physical address as the device only knows about phys addesses as it has no idea about CPU's MMU).

@MigthyMax

Is the value of BaseAddress 0? Or are you talking about the memory dump being 0?

If it's the latter, then check whether the master and memory enable bits are set in the PCI_COMMAND register (enable I/O accesses too if you have an I/O BAR). You need to enable those bits in order to access the memory.

Hans