ftell() speed : newlib & clib2

Found lately , that when i use newlib's ftell() it is slow when used offten on files more than few MB.

Real world difference is that for example doing the same (open a 20mb .b3d file and do a lot of ftell() on it) take on my x5000 (with ssd/sata/etc) for about 40 seconds, while on icore5 it take for about 2-3 seconds.

I didn't write any test case at moment which can meassuer speed of ftell() , but what i do is made such a simple test case:



Now, create simple "test.txt" with something like "aa" inside, so test-case will not crash/fail.

And then, i build version for newlib, and for clib2, and take a snoopy log of it. And result are:

newlib:




clib2:



As can be seen, in newlib version there is 4 native call happens for that simple test case:

GetFileposition()
GetFileposition()
ChangeFileposition()
GetFileposition()


While for clib2, there is no such calls.

In both cases that simple test case works of course, and show size of text file in console , so ftell() definately works in clib2 too, just , it seems to use lot less functions ?

Now, when i use newlib's ftell in actuall app , i can see is log a gazilion of those "4 calls" repeated all the time, which, probably reassons for them being slow in result.

Is anyone can think about how to fix it in newlib ? Maybe some buffers should be disabled, or something..

Or maybe anyone can think of some simple test case in which we can measuer ftell() time for both clib2 and newlib ?

As another test, in the code of game where all those ftell() used, i changed it so file firstly readed to the memory fully, and only then used by fteels and stuff, like:



And speed now changed RADICALLY. I mean, its just a few seconds instead of a minute.

That mean, that ftell() are VERY SLOW. I mean, very. Do not know through, if it only newlib's one, or its filesystem's issue.. That can be checked only by some test-case which can meassure speed of ftell().

Maybe some buffering should be disabled , which is enalbed by default and which make it be that slow ?

@kas1e

Quote:
You should definitely tell the DOS guys, i.e. Colin Wenzel and Tony Wyatt.

@Trixie
If only i have some simple test case showing the problem :) And after thinking a bit more, its probably can be all filemanagment operations guilty : fseek(), fread() and ftell().

What happens if you do a rewind(fp) (or fseek(fp, 0, SEEK_SET) if that's not available) before calling ftell? I know this defeats the purpose, but I wonder how many DOS calls you get at the beginning of the file.

@kas1e

The first two GetFilePosition() and the ChangeFilePosition() calls are from newlib's ftell() implementation calling fflush() at the beginning, which flushes any buffered writes and ensures that the file position is the same as the position in the internal read/write buffer (in your code this leads to the ChangeFilePosition() call to move one byte backwards because one extra byte had been read into the read buffer).

Clib2 also caches the current file position which means that the GetFilePosition() calls can be avoided, however it also means that any code that may directly or indirectly change the file position needs to keep this cached value up to date and valid or it can lead to buggy behaviour.

FWIW I'm currently looking into updating newlib to cache the file position in a similar way, which as I said above would at least make the GetFilePosition() calls unnecessary.

Another difference between clib2 and newlib is that newlib uses the newer DOS API calls with large file support, which means that it will be faster with new style vector port file systems such as NGFS and ram-handler but slower with legacy packet based file systems such as FFS2 and to some extent SFS which use packet packet based I/O and in FFS2 case probably does not even support the 64-bit packets leading to the first packet failing with ERROR_ACTION_NOT_KNOWN and DOS having to use a legacy packet as fallback.

@salas00
Quote:

What make me most curious, is HOW slow is it , i mean, few seconds when its in the memory, and about minute (!) when its as usuall from a file for 20mb sized file

But if get rid of GetFilePosition(), then it will mean things will be x3 times faster (through, still not few seconds). If , of course, take in account that issue with slow speed is amount of calls, and not something in the code of calls.

I can't say at moment how fast clib2 version in compare with newlib, because didn't wrote any "real meassure" test case, so i only can compare snoopy output.

From another side, i see that each calls take a very little microseconds, which can't be issue with speed. Maybe issue is amount of calls in general, not their speed when they executes. I.e. time which DOS spend between 2 calls.

Quote:

Yeah, while it sounds good, it seems that at moment something wrong : i do all test on NGFS , and that is what show very-slow results.

For sake of tests, i create an SFS2 volume, transfer all the game there, and run from it -> the same slow behaviour. So i assume it can be not issue with filesystems but instead with newlib realisation of those ftell/fseek/fread/etc.

@Salas00
Tested today's newlib: sadly no changes. The same slow, and i can see that when all this reading happens, hdd-led blinks, so it doing something, just slow by some reassons.

But i do check "ftell_newlib" test case, to see, what native calls is used now, and yeah no more of those calls, its now like this:



But still things the same slow, and hdd always blinks , like it doing something with file, just all the time and doing so slow :(

When i run the same game from SFS2 partition, then holy-omiga, its VEEEERY SLOW. It take not just 100 seconds as with NGFS, but i wait 15 minutes when run from SFS2 and it even didn't finished up ! I just reboot as tired to wait. And while when i run it from NGFS, hdd led "blinks" , then when i run it from SFS2, hdd lef always red, like something very heavy happens.

Definately something wrong there, and i not sure if it newlib fault at all. Maybe it can be something about those new filesystem things maybe (i mean vector/dos-packet). Maybe DOS itself involved..

@kas1e

Well the fflush() call does make using ftell() in any kind of file i/o loop a bad idea, as it throws away any buffered reads (so they have to be read again unnecessarily) and if you are writing it is also bad. A better idea right now would be to keep track of the current file position yourself by how many bytes you've read/written.

I checked newlib 3.1.0 sources for ftell()/ftello() and they don't use fflush() any more so I will see if maybe the changes can be back-ported easily into our newlib.

Of course updating newlib.library to newlib 3.1.0 would be even better but that will probably take a lot of work.

@salas00
Do you think it is newlib's issue ? I mean, not DOS, not filsystem one , but fflush() ?

Strange why on NGFS it take 100 seconds, and on SFS2 the same i even didn't waiting finishing (but i wait 10-15 minutes).

ps. in my case i only use reading for sure.

@kas1e

In your Snoopy log the ChangeFilePosition() call is only taking 3 microseconds (that's 0.000003 seconds) so I don't think NGFS is the culprit.

Using SFS\2 on my Sam460 it's taking about 18 microseconds and with ram-handler 7 microseconds.

@Salas00
Tested new newlib : yeah ! backporting of new fteel/etc from newlib 3.1.0 did the trick, now all fast !

now with usage of original code it take 19 seconds to load
with usage of change-to-memory-buffer it take 17 seconds to load
from sfs2 the same 19 seconds.

In other words you did it, thanks a bunch !

Through for release i of course still will use change-to-memory-buffer way, because when newlib will hit public no one know , but its good that you deal with anyway. I assume that will help a lot for other unix ports which rely on ftell() when readin/writing data.