DMA buffer for PCI busmaster headache

I want to allocate a DMA buffer for a device driver but I am completely lost how to accomplish this.

The PCI device wants to fetch commands from a command ring buffer located in main memory.
I have to provide the base address of this command buffer to the PCI device and the offset pointer where I wrote my last command. The PCI device will then fetch commands from this command buffer until its read offset pointer will match my write offset pointer.

I have allocated the command buffer with AllocMemTags in the following manner:



I wrote the address of pCmdBuf to the command buffer base address register inside the PCI device. Wrote the command to the next entry in pCmdBuf and updated the Write pointer register to match the location of this new command.
I can see that the DMA fetched something because his offset readpointer increased with 1 and matches my offset write pointer. But I do not get a response. This could mean that the DMA controller fetched some bogus command and chose to ignore it. I am sure that the command is valid. I am also sure that the response part of the PCI device is also setup correctly.

I did an experiment with disabling caches for this memory buffer in hypervisor mode but that didn't help. On top of that I acquired the physical addres of the buffer and wrote this to the PCI device command buffer base addres. This also didn't work.

I noticed a IExec funcions StartDMA/EndDMA. But from what I understand these are for devices that support scatter/gather functionality which this pci device doesn't support.

And there is also an older and depricated functions combinations called CachePreDMA/CachePostDMA/CacheClearE. This is supposed to flush caches and provide a physical address.

AFAIK these function are supposed to deal with the fact that the Amigaos4 memory architecture will not provide a contiguous buffer. But I provided the tags in AllocMemVecs to force contiguous memory block and both virtual and physical alignment. And since disabling caching didn't help either, I am right back at where I started this post: completely lost in what is the correct way to handle these kind of buffers. So I hope that someone can help me.

I also do not fully understand the PCI mechanism. Will the MMU handle DMA request to virtual address spaces? Or will a DMA access from a PCI device go to a physical memory address?

The PCI device expects little endian format so naturally everything is manually converted BE<-> LE by me in either direction when it is read/written through the PCI ibus. Only the functions in the PCIIFace for PCI config space are byte swapped automatically.

@geennaam

The address returned by IExec->AllocVecTags() is a virtual address as seen by the CPU.

To get the physical address of the memory you allocated you need to use the IMMU->GetPhysicalAddress() function from the exec.library "mmu" interface.

The function has to be called from supervisor mode so you will have to do something like:

@geennaam

In addition for the ring buffer you might want to allocate a certain number of memory pages (aligned set to page size, with size a multiple of page size) and use IMMU->GetMemoryAttrs()/IMMU->SetMemoryAttrs() to set the memory as MEMATTRF_CACHEINHIBIT.

You can get the mmu page size using IExec->GetCPUInfoTags().

@salass00

The experiment in my first post did exactly that:

1) Enter hypervisor and save context.

2) Flush caches

3) get/set memory attributes on buffer to inhibit caches ( CurrentAttr | MEMATTRF_CACHEINHIBIT).

3) get physical address of buffer and store in PCIe device DMA base register

4) Return to user space and restore context

5) Write the command to virtual buffer address. Physical address access will result in a DSI )

6) Signal pci device that the new command is available.

I don't know why but this didn't work for me. Still no response.


In the meantime I changed the sequence a bit:

1) Enter hypervisor and save context

2) Get physical address and store in PCI device DMA base register

3) Return to user space and restore context

4) Write command to virtual address

5) Flush cache to memory -> CacheCleanE() with CACRF_ClearD Flag set

6) Signal PCI device that the new command is available.

Now I receive an PCI interrupt when the response is available in another DMA buffer in memory. And this second buffer reads the response that I expected.

Strange that inhibiting caches for my buffer didn't do the trick. But at least it works now.

@geennaam

Inhibiting caches probably does not do an automatic flush of the affected region.

@geennaam

Using StartDMA()/EndDMA() would be a lot simpler. If the buffer you pass to it is physically contiguous, you'll only get one address in the returned list. So these functions don't require scatter/gather, they just support it.

Then you won't need to worry about supervisor mode, low level functions for cache flushing and getting physical addresses etc.

CachePreDMA()/CachePostDMA() are broken on OS4, so don't use them.

One other thing to watch out for: the DMAF_ReadFromRAM flag must be passed to StartDMA()/EndDMA() if the device will only read your buffer OR if it will read AND write your buffer. I.e. the only time it shouldn't be set is if you're sure the device will only write to the buffer.

@ncafferkey

So the only difference is that I do not need to perform a Cache flush manually?

What I do now is:
1. allocate a contiguous buffer which is physical aligned and is not allowed to be relocated with AllocVecTags(). This function returns the virtual address. (for OS4 access)

2. aqcuire the physical address in supervisor mode (for DMA controller access)

3. Perform a CacheClearE() after I've completed a write sequency to the memory region with the virtual address.

Anyways, that works like a charm now. But if it is more OS4 legal to do it with StartDMA()/EndDMA then I will switch of course.

@geennaam

I'd say both methods are valid.

What StartDMA()/EndDMA() will add to the game is hiding a bit more of the cache control logic and enabling you to more easily do scatter-gather on non-consecutive physical memory pages.