Catching memory corruption "in the act"

I'm dealing with some nasty memory corruption and I'd like to find the exact place where it happens. I have guard tags before and after the allocated memory, and I'm wondering if it'd be possible to protect these so I get a Grim Reaper when the program tramples on them.

Have you tried MemGuard?

@BSzili

What if you allocate a page sized (or is that cache line sized?) block in front and after the memory region (and alligned to its natural boundary) when you allocate it, and then use IMMU->SetMemoryAttrs() to make them MEMATTRF_READ_ONLY?

Haven't tried myself, though.

@Chris

Quote:
No longer works. Something changed in the OS around the release of the Final Edition, and MemGuard broke.

@trixie
Drats, that was I afraid of. That tool is quite old :(

@graff
That could work for the block in front of the allocation, but probably not for the one after it, as some padding has to be added for the alignment. There's a good chance the bogus write will happen in that region, especially if it has to be aligned on page boundaries.

@BSzili
You could try twice: once with aligned to page and protecting previous page to catch writes before the area. Then again aligining the end of area to end of page and protecting the next page to catch overflows. Maybe you can't catch both at the same time but this way could check both under and overflow in two tries.

@trixie

Quote:
What about it doesn't work?

I ask because I've been using MemGuard on FE (update 2) to test my programs and it's always seemed to work fine for me. It generates a "munglist" on demand, so it appears to be tracking allocations. It never found any issues, which I took to mean that my programs were clean; I never deliberately tried to overwrite memory to confirm that it was caught.

In that case I think I'll try both methods. Thanks for the tips!

@msteed

Quote:
I don't know, it just produces a lot of DSIs as soon as I start MemGuard. This is one an X5000, 4.1FE Update 2. What's your machine?

@BSzili

write a c++ class, and check min/max, allocators and dealocators can also be overloaded, so you know if memory is freed and so on.

Write a macros, they easily hide boiler plate checks, and are easy to disable, when they are not needed anymore.

@BSzili

I had a problem in Amos Kittens, there was large table containing a function table lookup, as some point in the code, it got corrupted. I solved the issue, by making a checksum, at different intervals check if the checksum did not match. This way, I figure out when corruption happened.

@BSzili

There different approaches you can do narrow down the problem..

disable functionality, until the corruption stops.

#if gfx_enabled
#endif

#if sound_enabled
#endif

you also allocate more memory then you need, if the corruption stops, you can kind guess what caused it.

Some times C++ can help you find bugs.. compile with all warnings..

"-wall"

Make sure index and pointers are initialized proper before use.

what if index I is not initialized.

And you do offset = pointer+I, you assume I is 0, but it can be any value.

It's a very large and complex project, think of 200,000+ lines of C++ code. This is why I need a stack trace at the place where it overwrites a buffer.

If it's originally from another OS I'd try to debug it there. If it does not seem to show up there it may be by chance, so try running through some debugging tools (valgrind?). Or if it's caused by some endianess bug, if possible try to run it in big endian Linux.

Also with gdb on this other OSes you can use watchpoints to catch memory writes to specific addresses.

The problem doesn't manifest in its natural habitat (Windows, little endian Linux). Big endian Linux is on my list to try, but only as a last resort as it runs very slow in QEMU :(

@trixie

Quote:
I definitely don't get any of those when I run it.

Quote:
Mine's an X1000, also 4.1FE Update 2 (no beta stuff). There are reputed to be kernel differences between the two that affect low-level debug-type functionality, such that some debug tools run on one but not the other. Perhaps this is one of those cases.

@trixie, msteed
This wall of DSIs and/or memguard hits on x5000 comes from console/shell stuff (at least on my setup). I reported that to TonyW saing that there are bugs in console components (as in the past any hit from memguard were real bugs in end). And i was told that memguard broken or x5k kernel broken, or so :)

@kas1e

Quote:
Plus, the MemGuard author says explicitly in the docs that the OS4 memory system keeps evolving, so it probably doesn't take much to break this kind of tool after a kernel change.

I normally use the CLIENT "ProgramName" option with MemGuard, so only the program I'm testing is monitored; that way I don't see any hits from other programs. (Of course, MemGuard still has to intercept all memory allocations and frees in order to check which program is making them, so if that's where the X5000 problem lies then CLIENT won't help.) I tried it briefly without the CLIENT option so it would monitor all programs; still no DSIs on the X1000.

The other thing you can try is building it against clib2 - it has a memory allocations debug mode (although it has been many years since I've tried it so can't remember exactly what it does)

I think you just need to add -D__MEM_DEBUG but there might be other stuff too.