[khmer] exceeding defined RAM limits?

[Michael R. Crusoe]

Hello Julia,

What version of khmer are you using?

That is, did you install via `pip` or a `git clone`?


On Tue, Dec 17, 2013 at 5:14 PM, C. Titus Brown <ctb at msu.edu> wrote:

> On Tue, Dec 17, 2013 at 04:36:34PM -0800, C. Titus Brown wrote:
> > On Tue, Dec 17, 2013 at 07:53:18PM +0000, Oh, Julia (NIH/NHGRI) [F]
> wrote:
> > Now, on to your real question :)
> >
> > > $python2.7 /home/ohjs/khmer/scripts/normalize-by-median.py -C 5 -k 20
> -N 4 -x 16e9 round2.unaligned.keep.abundfilt;
> > >
> > > I thought I would be maxing out at 64 GB ram for the hash table (I?ve
> also used 32e9), but I get the following RAM usage report of
> > >
> > > 4986693.biobos elapsed time:        23358 seconds
> > > 4986693.biobos walltime:         06:28:36 hh:mm:ss
> > > 4986693.biobos memory limit:       249.00 GB
> > > 4986693.biobos memory used:        249.76 GB
> > > 4986693.biobos cpupercent used:     98.00 %
> >
> > What the heck!? That's not supposed to happen!
> >
> > This is either a bug, or (most likely) is being caused by an
> overabundance of
> > high-abundance k-mers.  The latter is easy to fix -- I've filed a bug
> report to
> > fix the latter in the software overall [0] -- but would require you to
> modify
> > the script at the moment.  If you're up for that, put
> >
> >       ht.set_use_bigcount(False)
> >
> > at line 186 of normalize-by-median:
>
> Darn it, that can't be the problem; I just wrote a test against this
> behavior and we actually did things right in the script and ignored
> high abundance k-mers.
>
> So, this must be a bug of some sort.  Umm... Michael, any ideas?!
>
> cheers,
> --titus
> --
> C. Titus Brown, ctb at msu.edu
>
> _______________________________________________
> khmer mailing list
> khmer at lists.idyll.org
> http://lists.idyll.org/listinfo/khmer
>



-- 
Michael R. Crusoe: Software Engineer and Bioinformatician  mcrusoe at msu.edu
 @ the Genomics, Evolution, and Development lab; Michigan State University
http://ged.msu.edu/     http://orcid.org/0000-0002-2961-9670
@biocrusoe<http://twitter.com/biocrusoe>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/khmer/attachments/20131217/744e47f8/attachment-0001.htm>