<div dir="ltr">Hi Jens-Konrad,<div><br></div><div style>Sorry for the delayed response. (I was on vacation yesterday and hoping that someone more familiar with the partitioning code would answer.)</div><div style><br></div>
<div style>My understanding of the code is that decreasing the subset size will increase the number of partitions but will not change the overall graph coverage. Therefore, I would not expect it to lower memory requirements. (The overhead from additional partitions might raise them some, but I have not analyzed the code deeply enough to say one way or another about that.) As far as changing the number of threads goes, each thread does seem to maintain a local list of traversed k-mers (hidden in the C++ implementation) but I do not yet know how much that would impact memory usage. Have you tried using a fewer number of threads?</div>
<div style><br></div><div style>But, rather than guessing about causation, let's try to get some more diagnostic information. Does the script die immediately? (How long does the PBS job execute before failure?) Can you attach the output and error files for a job, and also the job script? What does</div>
<div style> qstat -f <job_id></div><div style>where <job_id> is the ID of your running job, tell you about memory usage?</div><div style><br></div><div style>Thanks,</div><div style> Eric</div><div style><br>
</div><div style><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Apr 8, 2013 at 3:34 AM, Jens-Konrad Preem <span dir="ltr"><<a href="mailto:jpreem@ut.ee" target="_blank">jpreem@ut.ee</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
I am having trouble with completing a partition-graph.py job.<br>
No matter the configurations It seems to terminate with error messages hinting at low memory etc. *<br>
Does LOWering the subset size reduce the memory use, what about LOWering the amount of parallel threads?<br>
The <a href="http://graafik.ht" target="_blank">graafik.ht</a> is 5.2G large, I had the script running as a PBS job with 240 GB RAM allocated. (That's as much as I can get it, maybe I'll have an opportunity in the next week to double it, but I wouldn't count on it).<br>
Is it expected for the script to require so much RAM, or is there some bug or some misuse by my part. Would there be any configuration to get past this?<br>
<br>
Jens-Konrad Preem, MSc., University of Tartu<br>
<br>
<br>
<br>
* the latest configuration after I thought on smaller subset size<br>
./khmer/scripts/partition-<u></u>graph.py --threads 24 --subset-size 1e4 graafik<br>
terminated with<br>
cannot allocate memory for thread-local data: ABORT<br>
<br>
<br>
______________________________<u></u>_________________<br>
khmer mailing list<br>
<a href="mailto:khmer@lists.idyll.org" target="_blank">khmer@lists.idyll.org</a><br>
<a href="http://lists.idyll.org/listinfo/khmer" target="_blank">http://lists.idyll.org/<u></u>listinfo/khmer</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div>Eric McDonald</div><div>HPC/Cloud Software Engineer</div><div> for the Institute for Cyber-Enabled Research (iCER)</div><div> and the Laboratory for Genomics, Evolution, and Development (GED)</div>
<div>Michigan State University</div><div>P: 517-355-8733</div></div>
</div>