AmigaOS 4 port of zstd V1.4.4

From www.zstd.net:
Quote:

So far I've compiled the libzstd static library and the zstd command-line program for AmigaOS 4.

If anyone wants to help with testing, the current zstd binary can be downloaded from:

https://www.dropbox.com/s/gxpjdsp97bvs ... td-1.4.4-20200307.7z?dl=0

There is no large file support as yet, as I haven't made the necessary changes the file I/O code.

My intention is to turn the static library into an AmigaOS shared library, as I have done with for example zlib, libzip, libpng and pixman.

@salass00

Hi, when runnign witour args from commandline, it shows an assignwedfe requester for
"stdin*\:" and "stdout*\:" .

"zstd --help" shows usage command/options without problem.

And as it have somekind of benchmark options:

####zstd -b
3#Synthetic 50% : 10000000 -> 3231410 (3.095), 1.78 MB/s , 49.5 MB/s
#

@jabirulo

Quote:

That's because of how dumb and lazy the code is for handling stdin input and stdout output. AFAICT the code looks for files called "/*stdin*" and "/*stdout*" on Linux/Windows/MacOS as well.

Just specify explicit input and output files and it should work correctly.

To get a list of the options available use "zstd -h".

To compress the file "testfile" to "testfile.zstd" you use:
zstd testfile -o testfile.zstd

To decompress it back you use:
zstd -d testfile.zstd -o testfile

I tested with compressing and decompressing a ~665MB AVI file on my Sam460 and the MD5 sums of the original and the decompressed file did match.

@salass00
tested with a plain textfile and it goes close to 7za compressor, very nice new (de)compressor!!!

#7za a test HD6750_slowVGAInit.txt

7-Zip (a) 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C,Utf16=off,HugeFiles=off,32 bits,1 CPU BE)

Scanning the drive:
1 file, 133052 bytes (130 KiB)

Creating archive: test.7z

Items to compress: 1


Files read from disk: 1
Archive size: 15246 bytes (15 KiB)
Everything is Ok


#zstd HD6750_slowVGAInit.txt -o test.zsdf
HD6750_slowVGAInit.txt : 13.34% (133052 => 17753 bytes, test.zsdf)
#


EDIT1: another test, this time with an ISO image:
#zstd KDH8:Sam460InstallCD-53.58.iso -o test.zstd
KDH8:Sam460InstallCD-53.58.iso : 57.02% (564692992 => 321987237 bytes, test.zstd)
#set
Process 6
RC 0
Result2 0
_LastCommand zstd
_LastCommandArgs KDH8:Sam460InstallCD-53.58.iso -o test.zstd
_RunTime 229.610135
..
#l
LICENSE 1530 ----rwed 04-Nov-19 17:54:32
test.zstd 321987237 ----rw-d 23-Nov-14 22:07:54
zstd 758064 ----rwed Hoy 07:38:20
#

@salass00

Nice! I've considered using zstandard to compress the firmware files that get embedded into the graphics drivers (the total size of which keeps growing). How easy was it to get the library to build?

Hans

@Hans

Quote:

Quite easy. The library source code is very portable so I didn't have to make any changes there.

I did write my own makefile because the included one was a bit messy and I like to have more control over what is done, but this didn't take too much time either.

The main issue I encountered was that my initial test with zstd CLI program generated a file after decompression that was different from the original one (much of the data had been replaced with 0xFEFECAFE).

My initial thought was that zstd doesn't correctly clear memory before using it, but after more testing I found the problem was with zstd's sparse mode and a bug in newlib.library that causes files to be extended with uninitialized data rather than zero bytes. I've fixed this by disabling the sparse mode if newlib.library < V53.59 is used.

@salass00

Neat.

Did that newlib bug already made it into the bug tracker?

@Raziel

It is already fixed in V53.59. It was such an obvious programming error that I'm surprised I hadn't noticed it while going over the code before.

The zstd.library now compiles but is as yet untested.

I was going to use the zstd CLI program for the first test but it uses some experimental functions which are only available with static linking so it's not suitable.

As libzstd uses a few global variables for debug levels and such I'm compiling it using the small data model and creating a separate copy of the data segment for each library opener. If this approach works well enough here, I might try to use it for other shared library adaptations in the future (if used in AmiSSL it would get rid of the baserel dependency which is one of the reasons it can't be compiled using a gcc newer than 4.0.4 right now).

Also while compiling the library using small data model I noticed that the .sdata section was much bigger than the eight bytes that I expected it would be from libzstd. This, it turned out, was because gcc was putting constant global data in the .sdata section instead of .rodata as usual. I've fixed this by adding the -mno-readonly-in-sdata option.