[khmer] exceeding defined RAM limits?

[C. Titus Brown]

On Tue, Dec 17, 2013 at 04:36:34PM -0800, C. Titus Brown wrote:
> On Tue, Dec 17, 2013 at 07:53:18PM +0000, Oh, Julia (NIH/NHGRI) [F] wrote:
> Now, on to your real question :)
> 
> > $python2.7 /home/ohjs/khmer/scripts/normalize-by-median.py -C 5 -k 20 -N 4 -x 16e9 round2.unaligned.keep.abundfilt; 
> > 
> > I thought I would be maxing out at 64 GB ram for the hash table (I?ve also used 32e9), but I get the following RAM usage report of 
> > 
> > 4986693.biobos elapsed time:        23358 seconds
> > 4986693.biobos walltime:         06:28:36 hh:mm:ss
> > 4986693.biobos memory limit:       249.00 GB
> > 4986693.biobos memory used:        249.76 GB
> > 4986693.biobos cpupercent used:     98.00 %
> 
> What the heck!? That's not supposed to happen!
> 
> This is either a bug, or (most likely) is being caused by an overabundance of
> high-abundance k-mers.  The latter is easy to fix -- I've filed a bug report to
> fix the latter in the software overall [0] -- but would require you to modify
> the script at the moment.  If you're up for that, put
> 
> 	ht.set_use_bigcount(False)
> 
> at line 186 of normalize-by-median:

Darn it, that can't be the problem; I just wrote a test against this
behavior and we actually did things right in the script and ignored
high abundance k-mers.

So, this must be a bug of some sort.  Umm... Michael, any ideas?!

cheers,
--titus
-- 
C. Titus Brown, ctb at msu.edu