<div dir="ltr">Hello Julia,<div><br></div><div>What version of khmer are you using?</div><div><br></div><div>That is, did you install via `pip` or a `git clone`?</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Tue, Dec 17, 2013 at 5:14 PM, C. Titus Brown <span dir="ltr"><<a href="mailto:ctb@msu.edu" target="_blank">ctb@msu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Tue, Dec 17, 2013 at 04:36:34PM -0800, C. Titus Brown wrote:<br>
> On Tue, Dec 17, 2013 at 07:53:18PM +0000, Oh, Julia (NIH/NHGRI) [F] wrote:<br>
</div><div class="im">> Now, on to your real question :)<br>
><br>
> > $python2.7 /home/ohjs/khmer/scripts/normalize-by-median.py -C 5 -k 20 -N 4 -x 16e9 round2.unaligned.keep.abundfilt;<br>
> ><br>
> > I thought I would be maxing out at 64 GB ram for the hash table (I?ve also used 32e9), but I get the following RAM usage report of<br>
> ><br>
> > 4986693.biobos elapsed time: 23358 seconds<br>
> > 4986693.biobos walltime: 06:28:36 hh:mm:ss<br>
> > 4986693.biobos memory limit: 249.00 GB<br>
> > 4986693.biobos memory used: 249.76 GB<br>
> > 4986693.biobos cpupercent used: 98.00 %<br>
><br>
> What the heck!? That's not supposed to happen!<br>
><br>
> This is either a bug, or (most likely) is being caused by an overabundance of<br>
> high-abundance k-mers. The latter is easy to fix -- I've filed a bug report to<br>
> fix the latter in the software overall [0] -- but would require you to modify<br>
> the script at the moment. If you're up for that, put<br>
><br>
> ht.set_use_bigcount(False)<br>
><br>
> at line 186 of normalize-by-median:<br>
<br>
</div>Darn it, that can't be the problem; I just wrote a test against this<br>
behavior and we actually did things right in the script and ignored<br>
high abundance k-mers.<br>
<br>
So, this must be a bug of some sort. Umm... Michael, any ideas?!<br>
<br>
cheers,<br>
<div class="HOEnZb"><div class="h5">--titus<br>
--<br>
C. Titus Brown, <a href="mailto:ctb@msu.edu">ctb@msu.edu</a><br>
<br>
_______________________________________________<br>
khmer mailing list<br>
<a href="mailto:khmer@lists.idyll.org">khmer@lists.idyll.org</a><br>
<a href="http://lists.idyll.org/listinfo/khmer" target="_blank">http://lists.idyll.org/listinfo/khmer</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><font face="courier new, monospace">Michael R. Crusoe: Software Engineer and Bioinformatician <a href="mailto:mcrusoe@msu.edu" target="_blank">mcrusoe@msu.edu</a><br>
@ the Genomics, Evolution, and Development lab; Michigan State University<br><a href="http://ged.msu.edu/" target="_blank">http://ged.msu.edu/</a> <a href="http://orcid.org/0000-0002-2961-9670" target="_blank">http://orcid.org/0000-0002-2961-9670</a> <a href="http://twitter.com/biocrusoe" target="_blank">@biocrusoe</a></font><br>
</div>
</div>