reduce memory footprint for loader

Especially for huge jobs, the grouper loader requires a lot of memory

From: Omaraie, Brad at UCLA
Sent: Thursday, August 15, 2013 7:07 PM
To: Chris Hyzer
Subject: GC overhead limit exceeded error

Hi Chris,
I was using a windows 7 64bit machine with 4G ram for testing the load to our grouper QA database before and load was successful. I was setting the maximum heap size to 3072m. I noticed when loading some of our bigger groups, the heap size would go up to 2.8g. Now I want to load the production data to our grouper prod database and we're using a 32 bit Linux machine with 3g of ram. The GSH session did not start on this machine when I was using 3072m as maximum size of heap. When I get a top from that machine, I see only 800k used memory and 2.2g free. But when I tried different maximum heap sizes, GSH session finally started with 1024m max heap size although there's more free memory available!

We started the load and now we are getting to some of those bigger groups. Looks like grouper tries to load all the member subject's for each group into memory first and then start the load to the DB. That's when it hits the 1024m size and crashes. I was wondering if you know of any setting to force grouper bring in smaller chunks of each group members into memory (not the entire group which in this case can go up to 450k member) before entering them to DB? Or maybe any suggestion on how we can increase the max heap size on Linux systems so the GSH session starts with higher max heap size?

Thanks
Brad

#################

1. 1. FROM CHRIS
      ##################

Can you switch to a 64 bit linux machine?

http://stackoverflow.com/questions/2457514/understanding-max-jvm-heap-size-32bit-vs-64bit

I noticed that we are loading the entire member objects (which are getting large), instead of just the subject_id and source_id (which is all that is generally used). We could have an option where it retrieves less data if you like...