[khmer] exceeding defined RAM limits?

[Oh, Julia (NIH/NHGRI) [F]]

Titus—thanks for the tip on variable coverage; will definitely try that out.

Michael—pretty sure I did a git clone. The last date in my directory is Sept 5th—but not sure if that would be pull date or your last modified date.

Thanks both!
Julia

On Dec 17, 2013, at 8:16 PM, Michael R. Crusoe <mcrusoe at msu.edu<mailto:mcrusoe at msu.edu>> wrote:

Hello Julia,

What version of khmer are you using?

That is, did you install via `pip` or a `git clone`?


On Tue, Dec 17, 2013 at 5:14 PM, C. Titus Brown <ctb at msu.edu<mailto:ctb at msu.edu>> wrote:
On Tue, Dec 17, 2013 at 04:36:34PM -0800, C. Titus Brown wrote:
> On Tue, Dec 17, 2013 at 07:53:18PM +0000, Oh, Julia (NIH/NHGRI) [F] wrote:
> Now, on to your real question :)
>
> > $python2.7 /home/ohjs/khmer/scripts/normalize-by-median.py -C 5 -k 20 -N 4 -x 16e9 round2.unaligned.keep.abundfilt;
> >
> > I thought I would be maxing out at 64 GB ram for the hash table (I?ve also used 32e9), but I get the following RAM usage report of
> >
> > 4986693.biobos elapsed time:        23358 seconds
> > 4986693.biobos walltime:         06:28:36 hh:mm:ss
> > 4986693.biobos memory limit:       249.00 GB
> > 4986693.biobos memory used:        249.76 GB
> > 4986693.biobos cpupercent used:     98.00 %
>
> What the heck!? That's not supposed to happen!
>
> This is either a bug, or (most likely) is being caused by an overabundance of
> high-abundance k-mers.  The latter is easy to fix -- I've filed a bug report to
> fix the latter in the software overall [0] -- but would require you to modify
> the script at the moment.  If you're up for that, put
>
>       ht.set_use_bigcount(False)
>
> at line 186 of normalize-by-median:

Darn it, that can't be the problem; I just wrote a test against this
behavior and we actually did things right in the script and ignored
high abundance k-mers.

So, this must be a bug of some sort.  Umm... Michael, any ideas?!

cheers,
--titus
--
C. Titus Brown, ctb at msu.edu<mailto:ctb at msu.edu>

_______________________________________________
khmer mailing list
khmer at lists.idyll.org<mailto:khmer at lists.idyll.org>
http://lists.idyll.org/listinfo/khmer



--
Michael R. Crusoe: Software Engineer and Bioinformatician  mcrusoe at msu.edu<mailto:mcrusoe at msu.edu>
 @ the Genomics, Evolution, and Development lab; Michigan State University
http://ged.msu.edu/     http://orcid.org/0000-0002-2961-9670    @biocrusoe<http://twitter.com/biocrusoe>