[khmer] Fwd: kmer counting stops at 248

C. Titus Brown ctb at msu.edu
Mon Aug 11 13:14:55 PDT 2014


We should probably create an issue about the vmem situation… not sure it’s fixable but I’m not sure I understand it either.

cheers,
—titus

Begin forwarded message:

> From: Alon philosof <aphilosof at gmail.com>
> Subject: Re: [khmer] kmer counting stops at 248
> Date: August 6, 2014 at 10:52:58 AM EDT
> To: titus at idyll.org
> 
> Hi,
> 
> the good news first - Khmer is working well now and the k-mer counts appear to be OK!
> 
> as for what caused the problems, I’m not quite sure. 
> when I originally wrote I was already using the latest branch.
> I realised at some point that the default python on the server was 2.6.6. I’ve tried another python version (2.7.8) but without success.
> finally, I’ve managed to install virtualenv (which I’ve failed at first) and from there it was smooth sailing.
> again, I think that for some reason memory allocation was wrong.
> another note, khmer seems to very hungry in Vmem. it’d be nice if it could be directed to prefer RAM or disk, as, at least in my case, Vmem is a limiting factor.
> thanks again,
> Alon 
> On Jul 27, 2014, at 2:15 PM, C. Titus Brown <ctb at msu.edu> wrote:
> 
>> On Sun, Jul 27, 2014 at 01:32:43PM +0300, Alon philosof wrote:
>>> Hello,
>>> i?ve came across a strange behaviour:
>>> I?ve installed Khmer 1.1 on a server running torque, since I don?t have root privileges I?ve downloaded the source and installed by: python setup.py install ?user.
>>> things seems to be working OK and I have attempted to count kmers and calculate histogram on a ~10gb metagenome file with:
>>> 
>>> .local/bin/load-into-counting.py --ksize 21 --threads 8 --min-tablesize 250e9 --n_tables 4 out.kh in.fastq
>>> 
>>> .local/bin/abundance-dist.py out.kh in.fastq out.hist
>>> 
>>> the resulting histogram shows counts up to only 248. I am running this with enough memory allocated and load-into-counting runs to completion without raising errors.
>>> in addition, I?d like to note that it runs on a single machine and not on multiple nodes.
>>> 
>>> however, when I run load-into-counting (--min-tablesize 64e8 --n_tables 4) on my local Ubuntu machine (32gb of ram)  running the same khmer 1.1 build  It runs to completion and calculates all kmers in the file.
>>> 
>>> when I abundance-dist.py on a .kh file from the server it again show counts to 248 only.
>>> 
>>> have you came across this behaviour before? any ideas how to solve that?
>> 
>> Hi Alon,
>> 
>> How strange -- nothing comes to mind immediately, I must admit :).
>> 
>> What happens when you run the tests? If you're in a virtualenv 'make test'
>> should work; if not, we have fixed a few problems in the latest master branch,
>> and you could run through our "getting started with development" docs
>> 
>> http://khmer.readthedocs.org/en/docs-hackathon/dev/getting-started.html
>> 
>> to use the latest branch.
>> 
>> My first guess is that one of the two machines is compiling in 32-bit mode.
>> If so, the tests will show that.
>> 
>> Thanks for any feedback you can give! This sort of thing worries me because
>> it hints at bigger potential problems :)
>> 
>> best,
>> --titus
>> -- 
>> C. Titus Brown, ctb at msu.edu
> 




More information about the khmer mailing list