Lead programmer: Sometime between Sunday and today 2.5 megs of memory appears to have gone missing. Could you look into it?
Me: That's very odd. Sure.
(later)
Me: I've examined the history logs in source safe and the only things that have changed are the new multistream irx file (which lives in IOP ram[0]) and a 400k increase in Lua memory. Just to make sure that it wasn't a fluke of memory aliasing[1] I locally reverted the changes and there wasn't a significant shift in allocations.
LP: Well, it would probably be a static allocation anyway, have you tried looking at the .map file[2]?
(later)
Me: There aren't any unusual allocations in the .map file, I found an older map file and did a diff, there aren't any significant differences.
LP: Well, the memory log of the crash is up on the network take a look at it...
(later after some confusion and idiocy on my part)
Me: Ah! Yes, I see the 2.5 meg difference. But in the one that my build generates, that 2.5 megs is still there.
LP: Well, that shouldn't be happening. It must be leaking memory, it would be really nice if we had a way to identify holes in memory when analyzing the logfile.
(much later after remembering how to code in Perl)
Me: Well, the good news is that there aren't any holes in the memory block list[3]. The end of memory is where it should be, but the start of memory is 2.5 megs too high, so if there's a leak it happened at the low end of the memory pool.
LP: Well the build that crashed is still running on a Tool[4], I suppose you could go look at the memory start value.
(we mosy over to a test rig[5] that has a yellow sticky note on both the monitor and the TV indicating that nobody is to use it because the crash is being analyzed)
Me: Well, the memory manager has the start memory position that matches the start memory position in the log file. Wait a second... this is a debug build[6].
LP: Oh, well that would explain where the 2.5 megs went. Well, sorry to have wasted your time on that.
Me: I think I'm going to go out for a walk.
--
[0] The PS2 has main memory, IOP memory, SPU memory, scratchpad memory, VU0 memory, VU1 memory, and VRAM. After a couple years of hard drinking it all becomes second nature.
[1] Historically there have been memory allocators which allocate powers of two and tack on a little bit to identify the block, so a theoretically nice allocation like char *foo = new char[256]; would actually allocate 512 bytes of RAM. This tends to get noticed when you allocate a meg and two megs go missing. I have yet to get bitten by this bug, but it took all of 10 seconds to check and if I hadn't checked it I'd be writing a blog entry on memory aliasing right now (I know how this Murphy's law thing works).
[2] The gnu linker (or in our case the ProDG linker) can be told to output a map of where everything is in your code. You can then take the (godawful) output and run it thorough a utility that sorts the data and spits out block sizes (writing the utility is left as an exercise for the coder). It's a very useful way to find things like: char modelReadBuf[ 8 * 1024 * 1024]; //todo, replace with dynamic allocation when I have time and we have some final art
[3] One of the things you do when writing a game is to write your own memory manager. It makes tracking leaks easier, it makes dealing with memory fragmentation easier, and it prevents you from getting pissed off at 4:00am and writing a memory manager that has bugs that will bite you in the ass for the rest of the project because you wrote it while you were overtired.
[4] Sony called them Tools. This is a technical term. Be aware that you're laughing at Sony (as opposed to laughing at me).
[5] One PC, one Television, one PS2 Tool. The PS2 Tool draws about as much current as the rest of the stuff combined. Half the people in our building have XBox kits as well. Our power bills would give you a heart attack.
[6] Our makefiles[7] (maintained by yours truly) have 5 working targets (debug, optimized, release, dvdrom, and dvdrom_final) depending on what level of optimization, debugging information, and media type the .elf should support. Debug mode contains no optimizations and for various reasons takes up significantly more memory than the other build types.
[7] For larger projects makefiles are a very effective solution, especially if you bother to learn how to really work with them.
Posted by matt at September 1, 2004 11:20 PM