[khmer] parition-graph memory requirements

[Eric McDonald]

Hi Jens-Konrad,

Sorry for the delayed response. (I was on vacation yesterday and hoping
that someone more familiar with the partitioning code would answer.)

My understanding of the code is that decreasing the subset size will
increase the number of partitions but will not change the overall graph
coverage. Therefore, I would not expect it to lower memory requirements.
(The overhead from additional partitions might raise them some, but I have
not analyzed the code deeply enough to say one way or another about that.)
As far as changing the number of threads goes, each thread does seem to
maintain a local list of traversed k-mers (hidden in the C++
implementation) but I do not yet know how much that would impact memory
usage. Have you tried using a fewer number of threads?

But, rather than guessing about causation, let's try to get some more
diagnostic information. Does the script die immediately? (How long does the
PBS job execute before failure?) Can you attach the output and error files
for a job, and also the job script? What does
  qstat -f <job_id>
where <job_id> is the ID of your running job, tell you about memory usage?

Thanks,
  Eric




On Mon, Apr 8, 2013 at 3:34 AM, Jens-Konrad Preem <jpreem at ut.ee> wrote:

> Hi,
> I am having trouble with completing a partition-graph.py job.
> No matter the configurations It seems to terminate with error messages
> hinting at low memory etc. *
> Does LOWering the subset size reduce the memory use, what about LOWering
> the amount of parallel threads?
> The graafik.ht is 5.2G large, I had the script running as a PBS job with
> 240 GB RAM allocated. (That's as much as I can get it, maybe I'll have an
> opportunity in the next week to double it, but I wouldn't count on it).
> Is it expected for the script to require so much RAM, or is there some bug
> or some misuse by my part. Would there be any configuration to get past
> this?
>
> Jens-Konrad Preem, MSc., University of Tartu
>
>
>
> * the latest configuration after I thought on smaller subset size
> ./khmer/scripts/partition-**graph.py  --threads 24 --subset-size 1e4
> graafik
> terminated with
> cannot allocate memory for thread-local data: ABORT
>
>
> ______________________________**_________________
> khmer mailing list
> khmer at lists.idyll.org
> http://lists.idyll.org/**listinfo/khmer<http://lists.idyll.org/listinfo/khmer>
>



-- 
Eric McDonald
HPC/Cloud Software Engineer
  for the Institute for Cyber-Enabled Research (iCER)
  and the Laboratory for Genomics, Evolution, and Development (GED)
Michigan State University
P: 517-355-8733
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/khmer/attachments/20130409/ed8df373/attachment-0002.htm>