NVMe device driver

First public release of the NVMe driver has been uploaded to os4depot.

From the readme:

What is NVMe:
NVMe is short for Non-Volatile Memory express. It's basically a transfer protocol for attachment of mass storage to a computer.
Unlike SATA, NVMe is specifically designed for solid state flash memory. There is no separate storage controller. Instead the NVMe SSD is directly connected to the PCIe bus through an M.2 slot.
The PCIe interface and NVMe protocol offer bandwidth and "IOs per seconds" which go way beyond anything that SATA can deliver. This high bandwidth is achieved because NVMe drives connect to a M.2 slots with 4 PCIe lanes. At the time of writing, the first PCIe 5.0 drives are emerging. These drives have a maximum interface speed of 16Gbyte/s while the actual peak performance is already approaching 10GByte/s.
Eventually, NVMe SSDs will replace SATA SSDs as the new industry standard. But since modern notebooks and PCs are not shipped with CD/DVD players (modern PC cases don't even have an external 5.25" bay anymore). And NVMe SSDs are rapidly becoming larger and cheaper. This will happen sooner rather then later.

How to attach NVMe to a NG system:
None of the current NG systems offer a M.2 slot directly on the mainboard. But since an NVMe M.2 slot is basically a PCIe x4 slot with a different form-factor, we can use a widely available passive PCIeM.2 adapter card.
These adapter cards are available with a PCIe x1 or PCIe x4 connector. Don't use the adapter cards with multiple M.2 slots because they will not work.

The X5000 has both x1 slots and a x4 slot available but there are some drawbacks:
- The x4 slot is blocked by a double slot GFX card. But with a PCIe extender cable for either the PCIe x4 slot or PCIe graphics cards, this slot becomes accessible.
- All PCIe slots, except the x16 Graphics card slot, are connected behind a PCIe switch. Which means additional latency.
- The PCIe slots are limited to version 2.0. This means an interface speed of "only" 500MB/s (x1) or 2000MB/s (x4). But either way this is still faster then the X5000 SATA2.0 ports. (300MB/s).

Which NVMe SSDs work with this driver:
This driver should work with any NVMe drive. It was developed to be compliant with the latest NVMe specification (2.0c).
A lot optional features are not implemented yet. Like the Host Memory Buffer for cacheless drives. But given the speed of our memory subsystem, I doubt that this will lead to any noticeable performance increase.
For now, the driver has been tested on a X5000 and in the x4 slot with the following drives:
- Samsung 970 EVO 1TB
- WD Black SN770 1TB

-----------------------------------------------------------------------
Installation:

Install the NVME hardware before adding the driver to the kicklayout

Copy "nvme.device" to sys:kickstart/ and add "MODULE Kickstart/nvme.device" to your sys:devs/kicklayout file.

Alternatively you can copy nvme.device" to DEVS: and mount your partitions manually with sys:system/Mounter. (Don't forget to set the tooltypes)

-----------------------------------------------------------------------
FAQ: (Foreseeable asked questions; This is a first release afterall )

Q: Yessss!! I will get >Gigabyte/seconds transfer speeds, right?
A: Erm, no. NVMe is designed for multithreaded and streaming access. Streaming as in: It takes some overhead to setup a transfer but then we get this train moving.
You will most likely use NGFS. This filesystem is single threaded and limits the maximum transfer internally to 32kbyte-128kb. The overhead needed to start those small transfers will kill speed in a single threaded environment.
The Amiga's slow message port system and the fact that it was decided that >AmigaOS3.9 drivers have to do gatekeeping for misaligned transfers is also not helping. (to say the least).
In short: Speed will be higher then a SATA drive with the same filesystem. But no miracles.

Q: So how fast is it?
A: With NGFS, barely faster than a SATA drive. SFS2 uses a maximum transfer size of 16MByte. I was able to measure speeds up to 384 MB/s while transfering a 1000MByte file between ram and a SFS2 partition. This beats SATA speeds but nowhere near its potential.
There is room for improvement but don't expect miracles.

Q: I don't believe you, I will play in media toolbox with blocksize, buffers, Maxtransfer and the Mask. Or?
A: Nope, these settings are completely ignored by PPC filesystems. So don't bother changing them. Apparently they are still there for legacy reasons only. Everything is decided by the filesystems for you.

Q: Wait a minute. Weren't modern SSD like NVMe SSDs supposed to drop 512 byte blocksizes and require 4096 instead?
A: That's an urban myth. NVMe drives require at least 4k host memory pages. The drive itself can support up to 64 different block sizes. The drive will tell which blocksize will give the best speed result. But so far, my Samsung 970 EVO supports 512 bytes only. The more modern WD black SN770 support 512 and 4k as well but is formatted as 512 bytes as factory default. Anyways, my driver will report back the formatted LBA to the filesystem.
In future updates I might provide a tool which allows you to change blocksize and reformat the drive accordingly. (Notice the "might")

Q: Ok, that is all disappointing. But I will be able to boot from this drive, right?
A: Wrong again. Currently, none of the NG systems support booting from NVMe drives. U-boot itself supports NVMe since 2017 with three isolated source files but I was told that it will take a complete rewrite of the X5000 U-boot to add support. In other words. Never.

Q: Then what is the purpose of this driver?
A: Next time, I will ask myself this question before I start the development.

Todo:

A lot. Like:
- A monitoring and control tool
- HMB for cacheless drives
- Support for more than one drive.
- .....

But I am not in a hurry.

Again....? Another driver so we can use new hardware in our beloved AmigaOne X5000. And this just after your AHI work. WoW!

Way to go @geennaam! Keep up the good work.

That is great news, thanks!
Pls, I cannot find it on os4depot.net, can you share direct link?

@sailor

It's in the upload queue.

@geennaam

thanks. I will test immediatelly in X1000 ( I know, the speed will be lower, .. )

great!!

@sailor

I wouldn't be surprise if it is faster. The PCIe slots are directly attached to the processor and we know that memory copies and PCIe performance on gfx cards are faster then the X5000.

@geennaam
I will see. PCIe is directly attached, but generation 1.0.

I have to upgrade a litle my X1000 - gfx card sometimes flashes and need new OS installation, is a little mess there .
Than I will do complex SSD tests, your NVMe + internal SATA2 + PCI SATA1

BTW. your driver cames at the right time - I have full all four SATA slots.
And I am run out of PCIe slots - x8 is now for linux graphics card.

Is possible split PCIe x8 to x4 + x4 without changes in CFE ? CPU has free root hub ( eight total ), but there is no option to do it in CFE.
Maybe will works somethnig like mentioned here. But for x8.

@sailor

Ow, 1.0 only. Even then, the difference isn't large. When I didn't install my WD Black SN770 properly, it fell back to 1.0 x1. Only peak performance with large files on SFS2 suffered from it.

It should always be possible to use an active PCIe expander to create additional PCIe slots. But I don't k ow the limitations of CFE and u-boot in this respect.

@geennaam
Quote:And while you are at it remove/disable diskcache.library.kmod in the kicklayout file, if you want to use SFS partitions.
It was implemented for spinning, magnetic based media like HDs only, but not for FLASH based media like SSDs, no matter if it's using SATA, NVMe, or some other access method.
Using diskcache.library on SSDs is very likely worse than not using it.

@joergQuote:

Thanks joerg, it is interesting information. Are there any more detailed info please?

I am interested in this questions:
- is better disable diskcache.library.kmod in all new systems with SATA SSD too? like Sam440, 460, X1000, X5000
- is better disable diskcache.library.kmod in all old systems with PCI-SATA SSD? like Micro, XE, Pegasos 2 - this the same like Sam440 becouse of the same SiI 3x1x controller.
- and what about if I have combination HDD + SSD in X1000 ? Is better to use diskcache?

I am using SFS everywhere.
And sorry, I am not found any documentation regarding this.

@geennaam

Thank you on your hard work!

@geennaam

Impressive ! we (as user) can only be proud to have people like you still developing new (and usefull) stuff for our beloved Amigas !

one question thought : can you boot from it ?

EDIT: nevermind I just find tha answer in the readme !

Q: Ok, that is all disappointing. But I will be able to boot from this drive,
right?
A: Wrong again. Currently, none of the NG systems support booting from NVMe
drives. U-boot itself supports NVMe since 2017 with three isolated source files
but I was told that it will take a complete rewrite of the X5000 U-boot to add
support. In other words. Never.

@Lio

you cannot boot, becouse you need to load kernel module.
But a little workaround might help to have all disks, includes System: on NVMe.

I am using on old NG machine this:
- on firmware readable partition ( Pegasos 2 IDE, CF card, FFS ) I have loader amigaboot.of and kernel modules ( /kickstart directory )
- on PCI SATA SSD I have System:, Work:, etc... all other disks

I think it will work the same way on X1000/X5000.
- amigaboot.of and kernel modules on SATA small partition, and rest on NVME...

Some quick benchmarks:

p50x0SATA.device: Samsung 840 SSD 120GB

Copy 1GB file from RAM: to NGFS/01: 51.5MB/s (20.4 sec)
Copy 1GB file from NGFS/01 to RAM : 98.9MB/s (10.6 sec)

nvme.device: Samsung 970 EVO SSD 1TB

Copy 1GB file from RAM: to NGFS/01: 129.5MB/s (8.1 sec)
Copy 1GB file from NGFS/01 to RAM : 152.0MB/s (6.9 sec)

Copy 1GB file from RAM: to SFS/02 : 327.7MB/s (3.2 sec)
Copy 1GB file from SFS/02 to RAM : 233.0MB/s (4.5 sec)

Now don't start using SFS/02 just yet because NGFS is a lot faster with small files. And then the speed difference between SATA and NVMe is a lot smaller (~25 - 50%).

Don't bother with diskspeed. This benchmark applications insists that my SATA drive is capable of >400MB/s. So it must be measuring some kind of cache memory.

@geennaam

Oooohhhh! That come as a big surprise indeed!
Another contribution that will turn out to be of fundamental importance once the PowerPC laptop based on the T2080 CPU will be ready!
Finger crossed, we are currently trying to run u-boot on the prototypes that proved to be electrically working.

@virgola

Then make sure that you fork a NVMe capable u-boot

@geennaam

yes, DiskSpeed is test of whole disk subsystem incl disk caches, so maximal results can be bigger than bus throughput.
SCSI speed is test of bus throughput - so it gives much lower results than DiskSpeed

I am using both for benchmarking. For example Pegasos 2 1.33 Ghz + AOS4.1fe + SATA SSD:
DiskSpeed Read average= 128 MB/s, Read max= 205 MB/s .. it is more than PCI and SATA1 limit
SCSISpeed Read average= 41 MB/s, Read max = 74 MB/s

@geennaam

This is pretty awesome and a great surprise. Thank you for all your hard work. It's a shame I was just made redundant otherwise I would go buy a drive and adapter...

Would be great if someone could do a video... kas1e?

Can I ask which NGFS you are using, the release version or the superior beta?

@daveyw

And why would you be redundant?

Yes, a newer NGFilesystem than shipped with Final Edition. Rock solid so far.

@geennaam

Quote:For example if you use it on a SFS partition with diskcache.library enabled the results you get are basically just the IExec->CopyMemQuick() speed ...

Use something like http://os4depot.net/?function=showfil ... /benchmark/scsispeed4.zip instead.