<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi,<br>
<br>
Thanks for investigating ! ;)<br>
<br>
<div class="moz-cite-prefix">Le 13/03/2013 16:54, Eric McDonald a
écrit :<br>
</div>
<blockquote
cite="mid:CAGhFaV1MW+jow_qZ53=SZPg8sxRvRUt8+Xk_Koek3QNJG=2zMg@mail.gmail.com"
type="cite">
<div dir="ltr">Thanks for the information, Alexis.
<div><br>
</div>
<div style="">The .below is an output file; it is empty because
the script crashed before anything was written to it.</div>
<div style=""><br>
</div>
<div style="">I have some more ideas about what may be
happening. But, rather than speculating wildly, I want to
reason in a slightly more data-driven manner. So, could you
please attach a copy of your job script, which has all of the
commands that you are running? (Among other things, I want to
see the parameters that you are using to construct the .kh
file.)</div>
</div>
</blockquote>
<br>
I attach the bash file with all the previous steps performed.<br>
<br>
<blockquote
cite="mid:CAGhFaV1MW+jow_qZ53=SZPg8sxRvRUt8+Xk_Koek3QNJG=2zMg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div style="">Also, can you please show us the output of the
following command:</div>
<div style=""> head -20 <span
style="font-family:monospace;font-size:10px">/mnt/var/home/ag/174r1_</span><span
style="font-family:monospace;font-size:10px">prinseq_good_bFr8.fasta.keep</span></div>
</div>
</blockquote>
<br>
Here it is :<br>
<br>
<tt>>ILLUMINA-2AC670:34:64CKCAAXX:2:1:3210:1066</tt><tt><br>
</tt><tt>GATNGGAAGAGCACACGTCTGAACTCCAGTCACATGTCAGAATCTCGTATGCCGTCTTCTGCTTGGAAAAGAAGA</tt><tt><br>
</tt><tt>>ILLUMINA-2AC670:34:64CKCAAXX:2:1:7734:1064</tt><tt><br>
</tt><tt>GATNGGAAGAGCACACGTCTGAACTCCAGTCACATGTCAGAATATCGTATGCCGTCTTCTGCTTGAAAAAAAAGT</tt><tt><br>
</tt><tt>>ILLUMINA-2AC670:34:64CKCAAXX:2:1:8363:1068</tt><tt><br>
</tt><tt>CGCNGAAGCATTTGTCGCACGGCTTGCGAAAGCAGGCGTGATCGCGCGCGATCCAAGATCGGAAGAGCACACGTC</tt><tt><br>
</tt><tt>>ILLUMINA-2AC670:34:64CKCAAXX:2:1:9951:1066</tt><tt><br>
</tt><tt>GATNGGAAGAGCATACGTCTGAACTCCAGTCACATGTCAGAATCTCGTATGCCGTCTTCTGCTTGGAGCACACGT</tt><tt><br>
</tt><tt>>ILLUMINA-2AC670:34:64CKCAAXX:2:1:13354:1070</tt><tt><br>
</tt><tt>GATNGGAAGAGCACACGTCTGAACTCCAGTCACATGTCAGAATATCGTATGCCGTCTTCTGCTTGAAAAAAAAAA</tt><tt><br>
</tt><tt>>ILLUMINA-2AC670:34:64CKCAAXX:2:1:14899:1066</tt><tt><br>
</tt><tt>GATNGGAAGAGCACACGTCTGAACTCCAGTCACATGTCAGAATCTCGTATGCCGTCTTCTGCTTGGAGCACACGT</tt><tt><br>
</tt><tt>>ILLUMINA-2AC670:34:64CKCAAXX:2:1:15273:1065</tt><tt><br>
</tt><tt>GATNGGAAGAGCACACGTCTGAACTCCAGTCACATGTCAGAATCTCGTATGCCGTCTTCTGCTTGGAGCACAAGT</tt><tt><br>
</tt><tt>>ILLUMINA-2AC670:34:64CKCAAXX:2:1:17148:1066</tt><tt><br>
</tt><tt>GTCNAGAACCTCGCGAGCTCGCCGGCGTTCTACGAGAAGCTTGGATTCACCGTCTTCGGGGGAAATGCCTCACAA</tt><tt><br>
</tt><tt>>ILLUMINA-2AC670:34:64CKCAAXX:2:1:5125:1074</tt><tt><br>
</tt><tt>GATCGGAAGAGCACACGTCTGAACTCCAGTCACATGTCAGAATCTCGTATGCCGTCTTCTGCTTGGAAAAAACGT</tt><tt><br>
</tt><tt>>ILLUMINA-2AC670:34:64CKCAAXX:2:1:5173:1073</tt><tt><br>
</tt><tt>CTGCTTCGCGCACTTATTTGCAGGGCCGATTTCGGCAGTCAGATCTGAATGAGGATTTGCTGCGCTACCTGGTTC</tt><br>
<br>
I also attach the filter-below-abund.py file with the following
changes<br>
- I have added<font face="Courier New"> #! /usr/bin/env python<br>
</font>- CUTOFF = 100 (since I diginormed to C=20)<br>
and I change the permisssions from 644 to 755 in the sandbox
directory<font face="Courier New"><br>
</font><br>
Alexis<br>
<br>
<blockquote
cite="mid:CAGhFaV1MW+jow_qZ53=SZPg8sxRvRUt8+Xk_Koek3QNJG=2zMg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div style=""><br>
</div>
<div style="">
Thanks,</div>
<div style=""><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Mar 13, 2013 at 10:48 AM,
Alexis Groppi <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:alexis.groppi@u-bordeaux2.fr" target="_blank">alexis.groppi@u-bordeaux2.fr</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Another clue : <br>
<br>
A empty .below file is generated
(174r1_prinseq_good_bFr8.fasta.keep.below)<br>
<br>
Alexis<br>
<br>
<br>
<div>Le 13/03/2013 15:13, Alexis Groppi a écrit :<br>
</div>
<div>
<div class="h5">
<blockquote type="cite"> Hi,<br>
<br>
<div>Le 13/03/2013 14:12, Eric McDonald a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Alexis,
<div><br>
</div>
<div>First, let me say thank you for being
patient and working with us in spite of all
the problems you are encountering.</div>
</div>
</blockquote>
<br>
That's bioinformatician life ;)<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>With regards to the floating point
exception, I see several opportunities for a
division-by-zero condition in the threading
utilities used by the script. These
opportunities exist if an input file is empty.
(The problem may be coming from another place,
but this would be my first guess.) What does
the following command say:</div>
<div><br>
</div>
<div> ls -lh <span
style="font-family:monospace;font-size:10px">/scratch/ag/khmer/</span><a
moz-do-not-send="true"
href="http://174r1_table.kh/"
style="font-family:monospace;font-size:10px"
target="_blank">174r1_table.kh</a> <span
style="font-family:monospace;font-size:10px">/mnt/var/home/ag/174r1_</span><span
style="font-family:monospace;font-size:10px">prinseq_good_bFr8.fasta.keep</span></div>
</div>
</blockquote>
<br>
The result : (the files are not empty)<br>
<tt>-rw-r--r-- 1 ag users 299M 12 mars 20:54
/mnt/var/home/ag/174r1_prinseq_good_bFr8.fasta.keep</tt><tt><br>
</tt><tt>-rw-r--r-- 1 ag users 141G 12 mars 21:05
/scratch/ag/khmer/<a moz-do-not-send="true"
href="http://174r1_table.kh" target="_blank">174r1_table.kh</a></tt><br>
<br>
<blockquote type="cite">
<div dir="ltr"><br>
<div>Also, since you appear to be using TORQUE
as your resource manager/batch system, could
you please attach the complete output and
error files for the job? (These files should
be of the form <job_name>.o2693 and
<job_name>.e2693, where <job_name>
is the name of your job. There may only be one
or the other of these files, depending on site
defaults and whether you specified "-j oe" or
"-j eo" in your job submission.)<br>
</div>
</div>
</blockquote>
<br>
I re run the job since I have deleted previous
(2693) err/out files.<br>
Here is the new file (merged with the option -j oe
in the bash script) :<br>
<br>
<tt>#############################</tt><tt><br>
</tt><tt>User: ag</tt><tt><br>
</tt><tt>Date: Wed Mar 13 14:59:21 CET 2013</tt><tt><br>
</tt><tt>Host: <a moz-do-not-send="true"
href="http://rainman.cbib.u-bordeaux2.fr"
target="_blank">rainman.cbib.u-bordeaux2.fr</a></tt><tt><br>
</tt><tt>Directory: /mnt/var/home/ag</tt><tt><br>
</tt><tt>PBS_JOBID: 2695.rainman</tt><tt><br>
</tt><tt>PBS_O_WORKDIR: /mnt/var/home/ag</tt><tt><br>
</tt><tt>PBS_NODEFILE: rainman</tt><tt><br>
</tt><tt>#############################</tt><tt><br>
</tt><tt>#############################</tt><tt><br>
</tt><tt>Debut filter-below-abund: Wed Mar 13
14:59:21 CET 2013</tt><tt><br>
</tt><tt>starting threads</tt><tt><br>
</tt><tt>starting writer</tt><tt><br>
</tt><tt>loading...</tt><tt><br>
</tt><tt>... filtering 0</tt><tt><br>
</tt><tt>/var/lib/torque/mom_priv/jobs/<a
moz-do-not-send="true"
href="http://2695.rainman.SC" target="_blank">2695.rainman.SC</a>:
line 49: 54757 Floating point exception(core
dumped) ./khmer-BETA/sandbox/fi</tt><tt><br>
</tt><tt>lter-below-abund.py /scratch/ag/khmer/<a
moz-do-not-send="true"
href="http://174r1_table.kh" target="_blank">174r1_table.kh</a>
/mnt/var/home/ag/174r1_prinseq_good_bFr8.fasta.keep</tt><tt><br>
</tt><tt><br>
</tt><tt>real 3m54.873s</tt><tt><br>
</tt><tt>user 0m0.085s</tt><tt><br>
</tt><tt>sys 2m2.180s</tt><tt><br>
</tt><tt>Date fin: Wed Mar 13 15:03:15 CET 2013</tt><tt><br>
</tt><tt>Job finished</tt><br>
<br>
Thanks again for your help :)<br>
<br>
Alexis<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div> </div>
<div><br>
</div>
<div>Thanks,</div>
<div> Eric</div>
<div><span
style="font-family:monospace;font-size:10px"><br>
</span></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Mar 13, 2013 at
5:38 AM, Alexis Groppi <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:alexis.groppi@u-bordeaux2.fr"
target="_blank">alexis.groppi@u-bordeaux2.fr</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Hi
Eric,<br>
<br>
Thanks for your answer.<br>
But unfortunately, after many attempts I'm
getting this error :<tt><br>
<br>
</tt><tt>starting threads</tt><tt><br>
</tt><tt>starting writer</tt><tt><br>
</tt><tt>loading...</tt><tt><br>
</tt><tt>... filtering 0</tt><tt><br>
</tt><tt>/var/lib/torque/mom_priv/jobs/<a
moz-do-not-send="true"
href="http://2693.rainman.SC"
target="_blank">2693.rainman.SC</a>:
line 46: 63657 Floating point
exception(core dumped)
./khmer-BETA/sandbox/filter-below-abund.py
/scratch/ag/khmer/<a
moz-do-not-send="true"
href="http://174r1_table.kh"
target="_blank">174r1_table.kh</a>
/mnt/var/home/ag/174r1_prinseq_good_bFr8.fasta.keep</tt><tt><br>
</tt><tt><br>
</tt><tt>real 3m30.163s</tt><tt><br>
</tt><tt>user 0m0.088s</tt><br>
<br>
Your opinion ?<br>
<br>
Thanks<br>
<br>
Alexis<br>
<br>
<br>
<div>Le 13/03/2013 00:55, Eric McDonald a
écrit :<br>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">Hi Alexis,
<div><br>
</div>
<div>One way to get the
'bleeding-edge' branch is to
clone it into a fresh directory;
for example:</div>
<div> git clone <a
moz-do-not-send="true"
href="http://github.com/ged-lab/khmer.git"
target="_blank">http://github.com/ged-lab/khmer.git</a>
-b bleeding-edge khmer-BETA</div>
<div><br>
</div>
<div>Assuming you already have a
clone of the 'ged-lab/khmer'
repo, then you should also be
able to do:</div>
<div> git fetch origin</div>
<div> git checkout bleeding-edge</div>
<div>Depending on how old your Git
client is and what its defaults
are, you may have to do the
following instead:</div>
<div> git checkout --track -b
bleeding-edge
origin/bleeding-edge</div>
<div><br>
</div>
<div>Hope this helps,</div>
<div> Eric</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue,
Mar 12, 2013 at 11:32 AM, Alexis
Groppi <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:alexis.groppi@u-bordeaux2.fr"
target="_blank">alexis.groppi@u-bordeaux2.fr</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000"
bgcolor="#FFFFFF"> <br>
<div>Le 12/03/2013 16:16, C.
Titus Brown a écrit :<br>
</div>
<div>
<blockquote type="cite">
<pre>On Tue, Mar 12, 2013 at 04:15:05PM +0100, Alexis Groppi wrote:
</pre>
<blockquote type="cite">
<pre>Hi Titus,
Thanks for your answer
Actually it's my second attempt with filter-below-abund.
The first time, I thought the problem was coming from the location of my
<a moz-do-not-send="true" href="http://table.kh" target="_blank">table.kh</a> file : in a storage element with poor level performance of I/O
I killed the job after 24h, moved the file in a best place and re run it
But with the same result : no completion after 24h
Any Idea ?
Thanks
Cheers From Bordeaux :)
Alexis
PS : The command line was the following :
./filter-below-abund.py <a moz-do-not-send="true" href="http://174r1_table.kh" target="_blank">174r1_table.kh</a> 174r1_prinseq_good_bFr8.fasta.keep
Is this correct ?
</pre>
</blockquote>
<pre>Yes, looks right... Can you try with the bleeding-edge branch, which now
incorporates a potential fix for this issue?</pre>
</blockquote>
</div>
From here : <a
moz-do-not-send="true"
href="https://github.com/ged-lab/khmer/tree/bleeding-edge"
target="_blank">https://github.com/ged-lab/khmer/tree/bleeding-edge</a>
?<br>
or <br>
here : <a
moz-do-not-send="true"
href="https://github.com/ctb/khmer/tree/bleeding-edge"
target="_blank">https://github.com/ctb/khmer/tree/bleeding-edge</a>
?<br>
<br>
Do I have to make a fresh
install ? and How ?<br>
Or just replace all the
files and folders ?<br>
<br>
Thanks :)<br>
<br>
Alexis
<div>
<div><br>
<br>
<blockquote type="cite">
<pre>thanks,
--titus
</pre>
<blockquote
type="cite">
<pre>Le 12/03/2013 14:41, C. Titus Brown a ?crit :
</pre>
<blockquote
type="cite">
<pre>On Tue, Mar 12, 2013 at 10:48:03AM +0100, Alexis Groppi wrote:
</pre>
<blockquote
type="cite">
<pre>Metagenome assembly :
My data :
- original (quality filtered) data : 4463243 reads (75 nt) (Illumina)
1/ Single pass digital normalization with normalize-by-median (C=20)
==> file .keep of 2560557 reads
2/ generated a hash table by load-into-counting on the .keep file
==> file .kh of ~16Go (huge file ?!)
3/ filter-below-abund with C=100 from the two previous file (<a moz-do-not-send="true" href="http://table.kh" target="_blank">table.kh</a>
and reads.keep)
Still running after 24 hours :(
Any advice to speed up this step ? ... and the others (partitionning ...) ?
I can have an access to a HPC : ~3000 cores.
</pre>
</blockquote>
<pre>Hi Alexis,
filter-below-abund and filter-abund have occasional bugs that prevent them
from completing. I would kill and restart. For that few reads it should
take no more than a few hours to do everything.
Most of what khmer does cannot easily be distributed across multiple chassis,
note.
best,
--titus
</pre>
</blockquote>
<pre>--
</pre>
</blockquote>
</blockquote>
<br>
</div>
</div>
<span><font color="#888888">
<div>-- <br>
<img
src="cid:part17.03070801.01070602@u-bordeaux2.fr"
border="0"></div>
</font></span></div>
<br>
_______________________________________________<br>
khmer mailing list<br>
<a moz-do-not-send="true"
href="mailto:khmer@lists.idyll.org"
target="_blank">khmer@lists.idyll.org</a><br>
<a moz-do-not-send="true"
href="http://lists.idyll.org/listinfo/khmer"
target="_blank">http://lists.idyll.org/listinfo/khmer</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div>Eric McDonald</div>
<div>HPC/Cloud Software Engineer</div>
<div> for the Institute for
Cyber-Enabled Research (iCER)</div>
<div> and the Laboratory for
Genomics, Evolution, and
Development (GED)</div>
<div>Michigan State University</div>
<div>P: <a
moz-do-not-send="true"
href="tel:517-355-8733"
value="+15173558733"
target="_blank">517-355-8733</a></div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
<span><font color="#888888">
<div>-- <br>
<img
src="cid:part21.05010102.08010501@u-bordeaux2.fr"
border="0"></div>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div>Eric McDonald</div>
<div>HPC/Cloud Software Engineer</div>
<div> for the Institute for Cyber-Enabled
Research (iCER)</div>
<div> and the Laboratory for Genomics,
Evolution, and Development (GED)</div>
<div>Michigan State University</div>
<div>P: <a moz-do-not-send="true"
href="tel:517-355-8733"
value="+15173558733" target="_blank">517-355-8733</a></div>
</div>
</div>
</blockquote>
<br>
<div>-- <br>
<img
src="cid:part23.01000303.01030509@u-bordeaux2.fr"
border="0"></div>
</blockquote>
<br>
</div>
</div>
<span class="HOEnZb"><font color="#888888">
<div>-- <br>
<img
src="cid:part24.04010209.05090502@u-bordeaux2.fr"
border="0"></div>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div>Eric McDonald</div>
<div>HPC/Cloud Software Engineer</div>
<div> for the Institute for Cyber-Enabled Research (iCER)</div>
<div> and the Laboratory for Genomics, Evolution, and
Development (GED)</div>
<div>Michigan State University</div>
<div>P: 517-355-8733</div>
</div>
</div>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<img src="cid:part25.00090909.05050501@u-bordeaux2.fr" border="0"></div>
</body>
</html>