NVMe device driver

@TearsOfMe

thanks! for HD is the same config like me, and with RadeonHD v3.7 it works

@sailor

Quote:
I'll contact you privately.

Hans

@Hans
Thanks!

Test of NVME in X1000, with different PCIe slots:
1. SATA2 SSD, internal sb600.device
2. NVME in PCIe x8 slot
3. in slot x8 is PCIe switch ( x8 -> x4 + x4 ), NVMe in first x4
4. NVMe in PCIe x1 slot

DiskSpeed: x8 is in average read 25% fastest, x1 has 7% slower maximal value
SCSISpeed: this test is very slow with PCIe switch ( the values are probably incorrect, but it signals that NVMe behaves differently with the switch ). And x1 has 5% slower maximal value.

For comparison with geennaam's X5000 ( his results here):
Copy1 GB file: x8 is fastest, x1 is slowest ( no big surprise here ).
But there is strange behavior with NVMe behind the switch: files can be freely copied to NVMe, but copying from NVMe behind the switch causes problems ( for bigger files there are no transport at all, so the result in graph is tested with 1 MB file ). The problems are the same with copy to RAM and copy to SATAdisk.

This switch is AXAGON PCEM2-ND, described as x8 -> x4 + x4. But in real there inside is Asmedia 2812 chip, which has upstream only x4. So strange behavior can be caused also with this. In any case - do not use this PCIe switch with NVMe ( not tested yet with other devices ).
In any case, I will test also adapter PEX8M2E2 from one server here , this one have chip ASM2824 with full x8 upstream.

@sailor

PCIe shouldn't be the limiting factor yet because you're still at only 1/3th of you PCIe1.0 x4 interface. So I would have expected the X1000 to be faster than the X5000 because of its altivec optimised copy routines and faster single core memory access.

I have a bit more time to work on the driver again so I will work on the reported issues and try to find some speedup opportunities.

You can also try to copy a combination of large and small files between ram and disk. For example the contents of the OS4 install CD.

Maybe, I should add a software polling fallback option in case interrupts don't show up after a timeout.

I still do not know how/if MSI/MSI-X is supported/implemented on OS4 side, so I have to rely on emulated interrupt pins. And it could be very well the case that some drives have issues with this old way of interrupt handling.

@Hans

Finding the MSI/MSI-X capability is not an issue. Support is mandatory in the NVMe spec and I have to enable them in the extended PCI capability registers.
I do have my own function to configure those interrupts.

But I do not know how I can use MSI/MSI-X interrupts from AmigaOS4 side. Currently I use MapInterrupt() and AddIntServer(). This works for emulated INT pin interrupts. But this doen't work for the multi vector capable MSI and MSI-X.

Although there seems to be some (premature?) MSI support in expansion/pci.h (SDK 54.16). There's no reference to these enums and structures or MSI/MSI-X in general in the (expansion library) autodocs.

@geennaam

I'm hazy on the details (it's been a while), but I think I was told that you don't get to enable them yourself. The OS enables MSI when/if it's supported. That's why I suggested checking if they were enabled.

Hans

@Hans

That would be a very weird situation. The OS4 can just randomly decide to enable any type of interrupt? They do not work in parallel so my code needs to do configuration as well which is different for each type of interrupt.

I can see that MSI/MSI-X is disabled. So for now just one emulated pin interrupt available.

And with the current aos4 implementation I don't see how they can be enabled anyways. Because the publically documented functions do not allow for multiple interrupt vectors for one PCIe device.

@geennaam

Quote:
The RadeonHD/RX drivers basically check if they're enabled and set a flag. The only difference is that the interrupt initialization code needs to set an RPTR_REARM flag. Aside from that, the GPU has its own mechanism for differentiating between the many interrupt sources (interrupt messages get written to a ring buffer).

You'd better talk to the kernel developers directly. You're probably the first person who needs the additional functionality that MSI/MSI-X provides.

Hans

@sailor
DiskSpeed results: Completely useless, especially when using SFS partitions with diskcache.library. In that case it's just the IExec->CopyMemQuick() speed, minus dos.library, file system, etc. overhead.
DiskSpeed is a bench marking tool to compare different file systems using the same driver/hardware, not for comparing different drivers/hardware.

SCSISpeed results: Useless if you are using the tiny default buffer/transfer sizes of it. Using something like "scsispeed BUF1=65536 BUF2=262144 BUF3=1048576 BUF4=16777216 ..." instead should generate more usable results for comparing MVMe with SATA.

C:Copy results: I don't know what the current default for it's BUF|BUFFER argument is (used to be 200 in old AmigaOS versions, thas's only 100KB on partitions with 512 bytes/block), but if it's still less than 32768 (16 MB): Useless as well...

@joerg
why not... but what benchmark is not useless?

@joerg

Is there an alternative to those programs you mentioned or are all benchmark tools useless?

Just wondering?

@sailor
SFS (original versions from John) included an own "diskspeed" implemetation from him. Obviously that "diskspeed" resulted in even more differences between FFS (extremely slow) and SFS (much faster than in normal usage) ...
Don't trust any statistics you didn't fake yourself

SCSISpeed is a benchmarking tool from 1989(!). While the OS4 port of it still (Edit:)can be used for comparing current drivers and hardware, that's only the case if you use the BUF1-BUF4 arguments to use much lager buffers/transfer sizes than the ones which were OK for 35 years old drivers and SCSI hardware.

@Raziel
Using SCSISpeed or C:Copy is still OK, just not with the very old, tiny default buffer sizes but using much larger BUF1-4 (SCSISpeed) or BUF|BUFFER (C:Copy) arguments suitable for current drivers and hardware instead.

My driver doesn't use transfer buffers except for alignment issues. It's a matter of passing pointers.

Not sure what's the benefit of using buffers for a c:copy at all when either source or destination is ram. So when I do a c:copy between ram and sfs, I can see that the large file is divided into transfers of 16MB. For both reads and writes.

@geennaam
I don't remember putting any transfer size limits in neither SFS nor diskcache.library, but maybe there is still something left in the original SFS sources I overlooked.

The BUF|BUFFER argument of C:Copy, using a multiple of the file system block size, i.e. "C:Copy BUFFER 2048 ..." is 1MB when using it on partitions with 512 bytes/block, is the size of reads and writes it's using.
Old AmigaOS versions used only 200 as default, 100KB with 512 bytes/block. Since you are getting 16MB parts it's very likely that the current default of C:Copy BUF|BUFFER is 32768 (*512 bytes/block = 16 MB).
With 16 MB reads/writes it's OK for comparing SATA with MVMe, but with the old 100 KB reads/writes it wouldn't have been.

RAM: isn't just ram, it has a file system/handler in between. Even with RAM: using very small reads/writes is much slower than using large ones.

Edit:
C:Copy BUFFER default is probably 16364 and not 32768, C:Copy should use the least common multiple of the source and destination block sizes, and RAM: has (or at least had 10-15 years ago) a "blocksize" of 1KB.
To make sure C:Copy results from different users are comparable the C:Copy version should be included. There should be no big difference between different C:Copy versions of AmigaOS 4.x, but the Enhancer Software C:Copy is something completely different, even if only the default for the BUFFER argument is different it can make a very big difference in the results.

@m3x

I've tested the WD Black SN770 with a PCI to PCIe adapter in my sam440-flex 800 MHz. Everything works as in my X5000. I can read/write sfs/02 partitions that were already on the drive successfully. Could also reinstall the drive and create new partitions. Read/write to the new partitions (Sfs2 and jxfs) showed no issues as well.

So the problems seems to be related to your drive.

I will prepare a probe later this week(end) and send it to your discord account.

@geennaam

Quote:

I have a spare adapter with PEX8111 which did not work with RadeonHD in my Sam. I'm sure you remember. Can I use it?

@mr2Quote:

of course! And let us know, if it works with NVMe.

But seriously, if your adapter in good state, chance of damage anything is the same like with any other card. Not zero, but very low.
But it will be allways slow - other PCI slots are 33 MHz there - roughly half of PCIe x1 speed.

@mr2

I've used the Pericom PI7C9X111SL adapter. Don't have the PEX8111 myself so I can't test it for you. But since it didn't work with the gfx card, I think you're better off using it as a doorstop.

But like Sailor already pointed out: There's no speed benefit over SATA1. Maybe it's even slower. A quick test showed similar read speeds but slower write speeds compared to my SATA HDD.

@sailor @geennaam

The reason is after few months, my new SSD disk started to crawl. OS boots in more than 2 mins and icons show like in a slideshow.
I have reinstalled old hdd.
I was thinking the SSD is not optimal for my Sam but on the other hand others use it succesfully. Probably I will stay with the old disk because I dont belive the PEX8111 would do the job.