Max. partition size in AmigaOS4.1 (SFS2 handler)

@geennaam
Quote:I don't understand why you even tried that, I wrote several times already, and of course it's mentioned in the SFS readmes as well, that SFS uses 32 bit block numbers resulting in a max. theoretical partition size of 2 TB with 512 bytes/block

Quote:TD_GETGEOMETRY64 didn't exist yet when I did the last updates of SFS, but filesystems don't use any values from TD_GETGEOMETRY anyway, unless LowCyl is 0 like on floppy disks for example, to support both DD (880 KB) and HD (1760 KB) floppies with a single DOSDriver.
On HDs, and anything similar like USB massstorage and NVMe with a RDB, the filesystems only use the values they get in their startup message from dos.library, which are the same CHS values as in the RDB created by MediaToolBox/HDToolBox/HDInstTool/etc.

Quote:How did you check the wrong 1.86TB size? The functions and data structures used by C:Info, the WorkBench, etc. are 32 bit as well and can't display anything > 2 TB correctly.

Quote:Using several partitions <= 2 TB (with 512 bytes/block) should work, but since I never had a HD > 500 GB, and the largest HD the original author of SFS had was only 10 or 20 GB, it could never be tested by a SFS developer.
Please test if it works if you disable diskcache.library in the kicklayout, maybe it's not SFS itself but diskcache.library not working correctly on drives > 2 TB.


@all
The last version of JXFS was intentionally read-only, to let users who used it copy their data to a partition with a working filesystem, without the risk of a JXFS version with write support destroying their data.
In case of JXFS it wasn't a license problem, Hyperion never had any. Beta versions of JXFS were just included as free contributions in some AmigaOS 4.x distributions.
It had too much bugs, and the bugs weren't just implementation bugs, but unfixable design bugs, and since Tony already started working on his better NGFS it didn't make sense to create a 3rd implementation of my JXFS.

@joerg

Quote:
Ah ok. I thought that you were saying that in the context of MediaToolbox. Anyways, a 1-2TB partition doesn't function properly on a >2TB drive. I can copy files and read back from it but the files are corrupted.

Quote:
That is not enrtirely correct. I do see filesystems sending the TD_GETGEOMETRY. Can't didn't watch closely if that is the case for SFS2 as well. But in any case, TD_GETGEOMETRY64 is not used except for mounter.library.

Quote:

Both the WB Disk info window and the WB drive root window top bar show the 3.72TB /xxxMB used correctly in case of NGF\01. In case of SFS2, the same windows shows 1.86TB. Of course I do not know if this is somehow an internal issue between SFS2 and WB which exist for SFS2 and not for NGF\01. But the partition doesn't function correctly anyway. In contrary to 1TB drives or NGF\01. This is at least a hint that somthing is wrong.

Quote:
From what I can tell, it doesn't. But these things happen when you don't have the tools at your disposal to check every corner of your software. I had to solve an issue with TD_GETGEOMETRY as well to make the >2TB drives get recognized correctly. Wouldn't have noticed the bug without having such a drive myself.

@geennaam
Quote:I wrote that the file systems don't use the results from it, not that they don't send the TD_GETGEOMETRY command.
It's used once on startup, but the results aren't used - unless LowCyl is 0. Only in this case the results from TD_GETGEOMETRY are used instead of the data from the file system startup message and additionally the TD_ADDCHANGEINT command is used to get information from the disk driver when a disk is removed from or inserted into the drive. On each disk change TD_GETGEOMETRY is used again to check if there is a disk in the drive or not and if yes use the data from TD_GETGEOMETRY result for the current disk. For example when replacing a DD floppy disk with a HD one in an Amiga floppy drive Sectors/Track changes from 11 to 22 and C*H*S * sector size from 880 KB to 1760 KB.

If LowCyl isn't 0, as it's the case on all partitioned drives, nothing from the TD_GETGEOMETRY result nor TD_ADDCHANGEINT is used but the CHS, etc. values from the file system startup message, the data from the RDB.

@joerg

Quote:

Ok understood. I'll stop looking into it further because I cannot fix SFS2 itself.

@geennaam

Too bad...

Does anyone know how far along NGFS is for non-X5000 machines?

@geennaam
Quote:Did you test it with and without diskcache.library?
If diskcache.library.kmod is installed SFS doesn't do the device I/O itself but just calls the diskache.library functions.
If diskcache.library.kmod is not installed the device I/O is done by SFS.
Would be very strange if the 64 bit device I/O code in SFS, which was implemented by John Hendrikx, has the same bugs for drives larger than 2 TB as the independent 64 bit device I/O code I implemented in diskcache.library.

@joerg

Quote:

Without diskcache of course. I only have SSDs.

But as final test I've also enabled diskcache in my kicklayout. Limit is the same. But at least now I can start the Spencer demo. So the truncated partition is usable.

Now this is really it. Sorry for those who are stuck with sfs2. But I am not going to spend more time on it.

@geennaam
It's strange that it doesn't work for you but does for Raziel
Quote:

Partitioning the disk to two 2048 GB and one with the rest works perfectly fine (two 2 TB and one 1.5 TB partition)

@joerg

Take my comment with a grain of salt...

I didn't do as much testing on the partitions as @geenam did.
i.e. i never copied anything to the partitions, i only created them to see in which size they came up in WB

@joerg

Reopening this topic as a result of the "Hyperion managed by liquidator; bankruptcy looming" thread.

Quote:

Ok, I am confused. I do not know what SP-engine is. All I know is that my NVMe driver replies to a geometry query with total number of blocks or CHS values that are valid for the size of the disk. But that MediaToolBox ends up with wrong values which results in the famous red dots as described above.


Quote:


As already mentioned in previous post, that is not what happens.

When using MediaToolbox, someone sends a SCSI "READ CAPACITY(10)" command. I'll reply in the 32bit "RETURNED LOGICAL BLOCK ADDRESS" field (and acc SCSI standard) the LBA of the Last logical block (amount of logical blocks-1). Someone translates this into an artifical CHS and this gets displayed in MediaToolBox

In case when the LBA of the last logic block exceeds 0xFFFFFFFE then I will reply this command with 0xFFFFFFFF (also acc. SCSI standard).

The SCSI standard describes that in case the returned value is 0xFFFFFFFF, the caller shall perform the SCSI "READ CAPACITY(16)" command. This is an extended command where the "RETURNED LOGICAL BLOCK ADDRESS" field is 64bit instead of 32bit. And that allows me the return the correct LBA of the last logical block for drives >2TB.

But the callers doesn't perform this SCSI "READ CAPACITY(16)" command when "READ CAPACITY(10)" returns 0xFFFFFFFF.

Instead the caller performs a TD_GetGeometry command. In this query I reply with actificial CHS values that multiply to the correct number of blocks that the >2TB drive supports because the field for the total amount of logical block in TD_GetGeometry is also limited to just 32bit.

Again, someone is doing translation of values returned by the driver and in this case my artificial CHS values are getting translated to some different CHS values that are wrong. These values result in the red dots as described above.
I need to ignore the red dots and manually correct the CHS values so that they multiply to the correct number of logical blocks. MediaToolBox still marks those values as invalid with those red dots but at least it allows me to create partititions that exceed 1.86TB. And those partitions work fine with NGFS.

But if MediaToolBox is just a GUI for "SP-engine" then "SP-engine" is the one that is the caller in my story? And the one creating those artificial CHS values afterall?


I do not intend to say that we need to change the way RDB stores the disk capacity. I said that we need to fix how the size is acquired from the disk in the first place. To prevent those red dots and offer a user friendly way to make >2TB disks work. Because not everybody knows what cylinders Heads and Sectors are and how they relate to the size of a disk.
As for storing capacity, CHS in the current RDB is perfectly fine for me.

You also mention mounter.library. But this library has its own method of aqcuiring disk size and that is TD_GetGeometry64. The returned structure contains a 64bit field for the amount of logical block. Iirc CHS is not even supported for this command

And now for 4k block size support.

Do we need to change the RDB format for that? Or any other components like dos.library? Because the both SCSI "READ CAPACITY (1x)" and TD_GetGeometry allow me to return the logical block size of 4k bytes. MediaToolBox picks up this value correctly (So I assume that "SP-Engine" is fine with them as well).
It's just that MediaToolBox doesn't allow me to create partitions. Everything is greyed out in the partitioning window.
If MediaToolBox is just a GUI then where is the limitation?

@geennaam
Quote:I'm probably mixing something and it's no longer used by MediaToolBox/SP-Engine but at least some of the old partitioning tools like HDToolBox (AmigaOS <= 3.1), HDToolBox (AmigaOS 3.5/3.9, different program) or HDInstTool didn't use READ CAPACITY(10) or even READ CAPACITY(16) but read SCSI mode pages with the disk size in CHS values instead.
The ATA drivers from sg2 include emulation for some SCSI mode pages for that.


Quote:Yes.
Quote:Not sure.


Quote:Theoretically not even if you want to use 4k RDB blocks, the RDB blocks (devices/hardblocks.h) include the size of the block: {type}_SummedLongs * 4.
Until AmigaOS 3.9 the min. size was 256 bytes, but due to some AmigaOS 4.x additions it's probably 512 bytes now. AFAIK there is no max. size, it just has to be a multiple of the physical block size and even if may not be required should be a power of 2 multiple.
But I guess at least some of the software working on RDBs only works with 512 bytes blocks, and that may include the firmware or SLB.

4k block sizes should only be required for partitions, not the RDB blocks, and that's possible with 512 byte RDB blocks as well: pb_Environment[] in struct PartitionBlock is struct DosEnvec (dos/filehandler.h) with de_SectorSize; /* in longwords: Physical disk sector size */.
The same struct DosEnvec is passed by dos.library to the filesystems, filesystems only use the data from that, they don't use something like TD_GetGeometry, SCSI commands or mode pages.
Requirements for that:
- The device driver needs to support reading/writing 512 bytes for creating/modifying the RDB, using read/modify/write 4k block for 512 byte writes if required, like for example 4k block HDDs do to support old software using 512 bytes accesses.
- The partitioning tool has to use a wrong rdb_BlockBytes of 512 instead of 4096 on 4k drives, to make other software reading the RDB but only supporting 512 bytes still work.
- The partitioning tool has to make sure the start and end of the partitions are on a 4k boundary on 4k block drives.


Since MediaToolBox/SP-Engine obviously has several bugs, did you try something else like HDInstTool, HDToolBox, etc. already?
Of course not usable for the drive you are booting AmigaOS from since that may need AmigaOS 4.x extensions, for example the SLB_v2 stored in the RDB will be deleted by AmigaOS <= 3.x tools, but should still work for other drives.