java work

Debugging an OutOfMemoryException

I've been hit by a nagging OutOfMemoryException

It happens in our "export" function where we scan the contents of a catalog and export data associated with its contents to XML files.

Each object that gets exported has a certain number of attached objects (parent/child relationship). What happens is that we scan each node of the catalog for parents and load the children and their associated data. Then we write that block to the file and forget about it (we even set the reference to null).

However, we need to remember each parent we saved so it gets saved only once even if it's in multiple nodes so we had a collection of integers which were the parents' ids that was used throughout the export process. We usually export a couple hundred or thousand parents.

We added an additional check on the children that get exported. So the collection was changed from a set to a Map. This map is indexed by the parent's ids and its value is a collection of integers which are the children's ids.

When we did this, the JVM crashed with an OutOfMemoryError, even when launching it with 512Mb of memory. It also seemed to be unable to fully garbage collect the memory on the server : everytime it peaked it didn't go back down to where it started (illogical since that map is not that huge and we don't keep all the parent/child data in memory...).

It turns out that the collection of child Ids is obtained from a method which is called on the parent object. This method calls the keySet() method on a map attribute which contained all the children.

When I changed this from a direct return of the keySet to generating a new set of Ids from the ids in the keySet() we no longer had the OutOfMemoryException. (ie doing something like result.add(new Integer(currentKeySetInteger().intValue()); inside an iterator loop). And the garbage collection implementation was able to reclaim all the memory every time it peaked to the max memory assigne to the JVM.

What we know is that the Integer instances used to index the child instances in the map attribute of the parent are being used in an object that stays the lenght of the export. So probably the child instance associated with them are not being garbage collected... but why ?
As I write this I am thinking that this is probably because the map index is actually the reference to the id attribute of the child object so the garbage collector really cannot reclaim it... tricky !! Humm... well, I just checked the code and no... that's not it : the id of the child is actually stored as a primitive int

So I'm stumped but it's a good lesson on garbage collection of how your JVM may starve off its memory if you're careless!

Posted by pgirolami76 at April 14, 2003 04:32 PM | TrackBack