[khmer] exceeding defined RAM limits?
Oh, Julia (NIH/NHGRI) [F]
julia.oh at nih.gov
Wed Dec 18 07:43:22 PST 2013
[ohjs at helix khmer]$ git checkout master
Branch master set up to track remote branch master from origin.
Switched to a new branch 'master'
[ohjs at helix khmer]$ make
===> lots of stuff, ending with:
copying build/lib.linux-x86_64-2.6/khmer/_khmermodule.so -> khmer
make[1]: Leaving directory `/home/ohjs/khmer/python'
[ohjs at helix khmer]$ git branch
bleeding-edge
* master
On Dec 18, 2013, at 8:10 AM, C. Titus Brown <ctb at msu.edu> wrote:
> On Wed, Dec 18, 2013 at 03:07:57AM +0000, Oh, Julia (NIH/NHGRI) [F] wrote:
>> Titus?thanks for the tip on variable coverage; will definitely try that out.
>
> Great -- should significantly improve sensitivity to low coverage "stuff"!
>
>> Michael?pretty sure I did a git clone. The last date in my directory is Sept 5th?but not sure if that would be pull date or your last modified date.
>
> OK, and then one last check... did you check out the 'master' or 'legacy'
> branch? What does 'git branch' report?
>
> To check out master, do:
>
> git checkout master
> make
>
> cheers,
> --titus
>
>> On Dec 17, 2013, at 8:16 PM, Michael R. Crusoe <mcrusoe at msu.edu<mailto:mcrusoe at msu.edu>> wrote:
>>
>> Hello Julia,
>>
>> What version of khmer are you using?
>>
>> That is, did you install via `pip` or a `git clone`?
>>
>>
>> On Tue, Dec 17, 2013 at 5:14 PM, C. Titus Brown <ctb at msu.edu<mailto:ctb at msu.edu>> wrote:
>> On Tue, Dec 17, 2013 at 04:36:34PM -0800, C. Titus Brown wrote:
>>> On Tue, Dec 17, 2013 at 07:53:18PM +0000, Oh, Julia (NIH/NHGRI) [F] wrote:
>>> Now, on to your real question :)
>>>
>>>> $python2.7 /home/ohjs/khmer/scripts/normalize-by-median.py -C 5 -k 20 -N 4 -x 16e9 round2.unaligned.keep.abundfilt;
>>>>
>>>> I thought I would be maxing out at 64 GB ram for the hash table (I?ve also used 32e9), but I get the following RAM usage report of
>>>>
>>>> 4986693.biobos elapsed time: 23358 seconds
>>>> 4986693.biobos walltime: 06:28:36 hh:mm:ss
>>>> 4986693.biobos memory limit: 249.00 GB
>>>> 4986693.biobos memory used: 249.76 GB
>>>> 4986693.biobos cpupercent used: 98.00 %
>>>
>>> What the heck!? That's not supposed to happen!
>>>
>>> This is either a bug, or (most likely) is being caused by an overabundance of
>>> high-abundance k-mers. The latter is easy to fix -- I've filed a bug report to
>>> fix the latter in the software overall [0] -- but would require you to modify
>>> the script at the moment. If you're up for that, put
>>>
>>> ht.set_use_bigcount(False)
>>>
>>> at line 186 of normalize-by-median:
>>
>> Darn it, that can't be the problem; I just wrote a test against this
>> behavior and we actually did things right in the script and ignored
>> high abundance k-mers.
>>
>> So, this must be a bug of some sort. Umm... Michael, any ideas?!
>>
>> cheers,
>> --titus
>> --
>> C. Titus Brown, ctb at msu.edu<mailto:ctb at msu.edu>
>>
>> _______________________________________________
>> khmer mailing list
>> khmer at lists.idyll.org<mailto:khmer at lists.idyll.org>
>> http://lists.idyll.org/listinfo/khmer
>>
>>
>>
>> --
>> Michael R. Crusoe: Software Engineer and Bioinformatician mcrusoe at msu.edu<mailto:mcrusoe at msu.edu>
>> @ the Genomics, Evolution, and Development lab; Michigan State University
>> http://ged.msu.edu/ http://orcid.org/0000-0002-2961-9670 @biocrusoe<http://twitter.com/biocrusoe>
>>
>
> --
> C. Titus Brown, ctb at msu.edu
More information about the khmer
mailing list