[khmer] exceeding defined RAM limits?

[C. Titus Brown]

On Wed, Dec 18, 2013 at 03:07:57AM +0000, Oh, Julia (NIH/NHGRI) [F] wrote:
> Titus?thanks for the tip on variable coverage; will definitely try that out.

Great -- should significantly improve sensitivity to low coverage "stuff"!

> Michael?pretty sure I did a git clone. The last date in my directory is Sept 5th?but not sure if that would be pull date or your last modified date.

OK, and then one last check... did you check out the 'master' or 'legacy'
branch?  What does 'git branch' report?

To check out master, do:

git checkout master
make

cheers,
--titus

> On Dec 17, 2013, at 8:16 PM, Michael R. Crusoe <mcrusoe at msu.edu<mailto:mcrusoe at msu.edu>> wrote:
> 
> Hello Julia,
> 
> What version of khmer are you using?
> 
> That is, did you install via `pip` or a `git clone`?
> 
> 
> On Tue, Dec 17, 2013 at 5:14 PM, C. Titus Brown <ctb at msu.edu<mailto:ctb at msu.edu>> wrote:
> On Tue, Dec 17, 2013 at 04:36:34PM -0800, C. Titus Brown wrote:
> > On Tue, Dec 17, 2013 at 07:53:18PM +0000, Oh, Julia (NIH/NHGRI) [F] wrote:
> > Now, on to your real question :)
> >
> > > $python2.7 /home/ohjs/khmer/scripts/normalize-by-median.py -C 5 -k 20 -N 4 -x 16e9 round2.unaligned.keep.abundfilt;
> > >
> > > I thought I would be maxing out at 64 GB ram for the hash table (I?ve also used 32e9), but I get the following RAM usage report of
> > >
> > > 4986693.biobos elapsed time:        23358 seconds
> > > 4986693.biobos walltime:         06:28:36 hh:mm:ss
> > > 4986693.biobos memory limit:       249.00 GB
> > > 4986693.biobos memory used:        249.76 GB
> > > 4986693.biobos cpupercent used:     98.00 %
> >
> > What the heck!? That's not supposed to happen!
> >
> > This is either a bug, or (most likely) is being caused by an overabundance of
> > high-abundance k-mers.  The latter is easy to fix -- I've filed a bug report to
> > fix the latter in the software overall [0] -- but would require you to modify
> > the script at the moment.  If you're up for that, put
> >
> >       ht.set_use_bigcount(False)
> >
> > at line 186 of normalize-by-median:
> 
> Darn it, that can't be the problem; I just wrote a test against this
> behavior and we actually did things right in the script and ignored
> high abundance k-mers.
> 
> So, this must be a bug of some sort.  Umm... Michael, any ideas?!
> 
> cheers,
> --titus
> --
> C. Titus Brown, ctb at msu.edu<mailto:ctb at msu.edu>
> 
> _______________________________________________
> khmer mailing list
> khmer at lists.idyll.org<mailto:khmer at lists.idyll.org>
> http://lists.idyll.org/listinfo/khmer
> 
> 
> 
> --
> Michael R. Crusoe: Software Engineer and Bioinformatician  mcrusoe at msu.edu<mailto:mcrusoe at msu.edu>
>  @ the Genomics, Evolution, and Development lab; Michigan State University
> http://ged.msu.edu/     http://orcid.org/0000-0002-2961-9670    @biocrusoe<http://twitter.com/biocrusoe>
> 

-- 
C. Titus Brown, ctb at msu.edu