How detect and fix heap trashing ?

While trying to deal with some code (its a game, which didn't writen by me, so i can't know what every line of code do), found that it crashes all the time when i navigate over menu. Just pressing up/down over menu items and bbah!

Crashlog always point out on some free(texdata) call. Of course first thing i tried to do is add check like if(textdata) free(texdata), but it still crashes. On other platforms (win32, linux), it didn't crashes there. So probabaly their OSes deal with overflows of this kind , dunno.

So, while stack trace always point out on free(texdata), that mean that probabaly some heap corruption happens somewhere because of some buffers overruns/overflow whatever which may have impact on the heap.

Now, how to detect which part cause issues, if there is lot of source code all over the place with mallocs , strcpy, and all kind of stuff which works with memory and can trash the heap (and, i not 100% sure its a heap trashing, but all looks like this).

For first i add -Wall and -O3 , so to catch all possible issues. And, that what i have in the menu.c (at least i hope it come from there):




So by warning it already looks like something overwrite buffers (which probabaly in end cause heap corruption => crash).

That relevant parts are:




Pure replaceing char str[256]; on char str[1024]; deal with warnings, but didn't deal with crash which mean heap still fuckedup.

Visually i even can see, when i go through menu, every switch can leave a little artefact of previous entry, which mean or it didn't clean correctly, or didn't filled correctly.


Now question is: how to found where. I mean some automatical way. Maybe undef all mallocs/free to something which will count how much and when allocatred and then freed and write it to console, so i can compare..

Any ideas ?

try:

char str[1024] = {0};

i remember well that if a variable is not initialized can cause bad stuff on os4.
Or try to create a pointer and use malloc/memset or calloc and see if something change

Those warnings simply come from the size of the "text" and "settingtext" attributes of the entry-struct.
3 bytes -> the ' : ' part of the format string, if both text and settingtext are empty strings.
between 4 -> 3 byes from above plus 0 terminator if both strings are empty.
and 514 -> if both text and settingtext are filled to their maximum plus 4 bytes from abobe. Apparently both are defined like this:

struct Entry {
char text[256];
char settingtext[256];
}

So, while those warnings are all nice, they don't necessarily mean too much.
Especially in older code you often find such constructs with fixed length string buffers and it is all just fine, if the code which uses them is correct. And those warnings don't tell you anything about whether its used correctly or not, it simply gets fired on every such construct.

So the warning only tells you a theoretical cause of trouble, nothing more. It most likely doesn't actually point you to your problem in this case here. Most likely everything is just fine with that warned code. I'd ignore it.
You could silence them by changing str[256] to str[514]. Of course it may still overflow if e.g. the entry-struct points into invalid memory ;)

Concentrate on the use of "texdata" first. Where is it created? Where is it freed? Is it freed at more locations than the crash-site? Is it set to NULL after free?

@afxgroup
Thanks, tryied at moment {0} thing : didn't help much.

@Daniel
Thanks for worry !:)

Creation of texdata and its freed done in one function:



That getStringPixmapFT take the font's letters via freetype (it take some dejavu.ttf from which got letters and build those textures showing in the menu)




I just put few prinfs around free(texdata), and run game, and it doing about 100-200 or more free(texdata) before crashes. And crashes always on some particular entry to which i switch second time (first time all ok).

I also note, that some menu entries have some trailed line over letters "p, g and y". And seems that once i second time come to the entry with such trailing line i crashes, see how it looks like visually:

(press open in new tab for fullsize)



What is interesting, in win32 version i have the same over those letters, but it didn't crash (but then, win32 probabaly catch all kind of illegal and bad memory accesses /frees and most of time you even didn't know that there is something bad happens).

Just in case there is crashlog as well:
Crashlog.txt

@kas1e
Please try the following:
in create_string_quad change this
char * texdata;
to this
char * texdata=0;

EDIT:
Ah, no (although: keep the above anyway, doesn't hurt).
Try the following:

Change the line
(*data)=malloc(w*h);
to
(*data)=malloc(w*h+1);

I have had weird things happen with Freetype (OS4). As you said, something that works at first, stops working. And does work after a reboot. Although I recently updated it (latest .so from OS4Depot), not long enough ago to tell, if it is all good.

@Daniel
Nope, still the same :(

@Thematic
I build all statically, and also have all latest from os4depot. And basically never have problems with freetype, its for sure something bad in game's code.

@kas1e

Does commenting out the line
my_draw_bitmap(...)
change anything in terms of crashes?

Hard to give you more hints here. I mean, e.g. the function "decode". Who knows what it does. Maybe it's buggy and makes n to become an out-of-bounds-index. Or str is invalid in the first place.
If you can rule this out then it's most likely some obscure side-effect of memory coruption somewhere else

@Daniel
With commenting out my_draw_bitmap() i can't see any text. Even if i press "Esc" for going to menu, there is none. Its even didn't looks like menu is here, game continues like nothing was pressed.

That "decode" thing are:



Will check if str is valid one as well..

I just thining before making the topic, maybe there is some general rule how to find those strange help corruption bugs. We on aos4 didn't have any normal stuff for checking in realtime, so maybe some static-analizators of C code can help .. Or adding of "dmalloc() kind" things which will count how many alloca/free done, how many each alloc/free, which pointers used to what addresses ,etc..

@kas1e
Of course you dont see any text if you comment it out.
The question was whether this makes the crashs go away or not.

@Daniel
It crashes only when i navigate in the menu, when i just play game it never crash. So as i didnt have menu when comnenting out my_draw_bitmap(), i cant triger the crash..

@kas1e

Please try DrMemory on Windows and Valgrind on Linux.

@Capehill
Tryed DrMemory, that one good! Through windows version builded without debug info, so all in ??? mostly, but just pure run and exit give me:



Need to build win32 version with debug symbols to be able to see at least where to look.

@kas1e

try this

@Daniel

I retested again with commented out my_draw_bitmap(), and sorry, was wrong, menu works indeed even if i don't see it. I can press 12 times "down" (so to be on entry "exit") and exit from the game.

So, i just tryied to reproduce crash when commented this my_draw_bitmap() out: and can't. With uncommenting it back, its enough for me to press "esc" for going to menu, then just press : 2 times down, 2 times up -> crash. With commented out my_draw_bitmap() i for sure have no crash, retested 10 times.

I can go up/down as much times as i want, and then go up, and press 12 times down to go to the "Exit", and exit fine, no crash.

So it something which want to be accessed from that function. *data ?

Whole my_draw_bitmap() are:




@Billyfish
Thanks, tried, that didn't help.

Through, should to note, its probabaly can be something around that, because i build win32 version with debug symbols, running DrMemory on it, and , it point out exactly on those lines like:



And that 287 line, are
Quote:

from the menu_create_textobj(). Same entry in the menu_entry_set_settingtext() didn't leak, only from menu_create_textobj().

Why i think its not false alarm, its because DrMemory found with the same "UNINITIALIZED READ: reading register eax" real bug which was fixed (but i do test on unfixed version, so to check if it will find it, and it is). So probabaly even if it will be in end not related to the crash, there sitll some issue in those functions. I will recheck if your replacement will bring no error by DrMemory

@Daniel
And i also check in DrMemory win32's logs more , and found another possible issue which may very well related to the fact that we didn't crash when comment out draw_my_bitmap() stuff (which needs *data):



And, line 308 of font.c , are in that getStringPixmapFT, and it's : (*data)=malloc(w*h) , which yes, we trying to swap on (*data)=malloc(w*h+1); before, but that didn't help to the crash as well :(

Also, it found a leak in texobj.c:



In function:



At line 93, which are obj=malloc(sizeof(textObj));

Not sure which one of those is guilty, but it 99% its all around of those functions

@all
In short summary to avoid reading last two big posts, DrMemory found:

1. one UNINITIALIZED READ in the menu_create_textobj(), at line if( entry->show_subsetting && entry->settingtext[0]!=0 ){

2. Also one leak in the getStringPixmapFT() , on the line (*data)=malloc(w*h); (and data there, is that textdata, on which we crash later when do free(textdata).

3. And another leak in the textObj_new() function on the obj=malloc(sizeof(textObj)); . And that textObj_new() are called from menu_create_textobj(), on which previously we have a leak too.

So, from menu.c we do menu_create_textobj() which have unitiialized read on if( entry->show_subsetting && entry->settingtext[0]!=0 ){ , then from same function we call after that leak textObj_new() , which also continue to have leak in the obj=malloc(sizeof(textObj)); , and then , create_string_quad() called, which call getStringPixmapFT() in which we already on big leak when do (*data)=malloc(w*h);

In end of which it probabaly fuck the heap, and give us that crashes on free(). Only to understand from where it come :)


All code of functions posted in previous comments, so while i step by step trying to figure what happens, maybe anyone will have a clue already.

Thanks !

@kas1e

Run the same code snippet in Windows or Linux, maybe you get lucky, and tells you what happens.

@Liveforit
Do you know who captain Picard is ? :) I do :)

@kas1e

I'm sci-fi nerd so yes. Always keeps my eyes open for good new series.