<div dir="ltr">Hi Alexis,<div><br></div><div style>I'm glad to hear that you were able to get past the problem. Admittedly, I am still somewhat puzzled by it.</div><div style><br></div><div style>As far as estimating hash table size is concerned, the Bloom filter uses 4 hash tables by default. The size of each hash table is the number of bytes specified with the "-x" parameter. So, if you chose 32e9 as the hash table size, then the entire Bloom filter would be 128 GB plus some additional overhead. But, I don't think you need to choose that large of a hash table in most cases. Somewhere (possibly in my inbox), Titus created a guide about what size hash table to generally use with certain kinds of data. If it is not already in the documentation, then it probably needs to be added.</div>
<div style><br></div><div style>With the scripts in the 'scripts' directory, you can use the '--help' option to find out what arguments are supported. "-x" and "-N" are the Bloom filter (hash tables) control parameters. (Note: Your memory usage is mostly dependent upon these and _not_ on the size of your input files. Your false positive rate is determined by the size of the Bloom filter and the number of unique k-mers in your data. The number of unique k-mers scales as a function of the size of the data set and the diversity of k-mers within the data. As mentioned in the previous paragraph, Titus has created a guide which accounts for part of this diversity (I believe) based on whether something is a particular kind of genome or a metagenome. But, this is more biology than I know and he can correct me if I misrepresented anything. :-) So, you have controls on your memory consumption already. The '--savehash' option works with the '-x' and '-N' options in 'normalize-by-median.py'. I hope this makes sense and addresses your comments - please let us know if it is still not clear.</div>
<div style><br></div><div style>Good luck with the partitioning! Hopefully that will go more smoothly for you.</div><div style><br></div><div style>Thanks for you working with us to debug the problem,</div><div style> Eric</div>
<div style><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Mar 15, 2013 at 11:05 AM, Alexis Groppi <span dir="ltr"><<a href="mailto:alexis.groppi@u-bordeaux2.fr" target="_blank">alexis.groppi@u-bordeaux2.fr</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Hi Eric,<br>
<br>
Good News : I may have found the solution of this tricky bug<br>
<br>
The bug come from the hastable construction with
load-into-counting.py<br>
We have used the following parameters : load-into-counting.py -k 20<b>
-x 32e9</b><br>
With -x 32e9 the hashtable grows until it reaches the maximum ram
available at the moment, independantly of the size of the fasta.keep
file.<br>
But, in a manner I ignore, this file is not correct.<br>
I realise this by repeating the two steps load-into-counting.py and
then filter-below-abund.py on the very small subsample of 100000
reads.<br>
==> It generates a <a href="http://table.kh" target="_blank">table.kh</a> of 248.5 Go (!) and leads to the same
error : Floating point exception(core dumped).<br>
<br>
I tried to performed these two steps on the whole data sets (~2.5
millions of reads) with load-into-counting.py -k 20<b> -x 5e7</b><br>
<br>
==> It works perfectly but I got a warning/error in the output
file :<br>
** ERROR: the counting hash is too small for<br>
** this data set. Increase hashsize/num ht.<br>
<br>
Finally I ran the two steps with load-into-counting.py -k 20 <b>-x
1e9</b>.... And It works perfectly ! in a fews minutes (~6 mins)
without any warning or error.<br>
<br>
In my opinion, it will be useful( if possible) to include a control
on the hastable creation by the script load-into-counting.py.<br>
By the way, how this is managed via normalize-by-median.py and the
--savehash option ?<br>
<br>
Now shifting to the next steps (partitioning). I hope in a more easy
way ;)<br>
<br>
Thanks again for your responsiveness.<br>
<br>
Have nice Weekend<br>
<br>
Alexis<br>
<br>
<div>Le 15/03/2013 02:02, Eric McDonald a
écrit :<br>
</div><div class="im">
<blockquote type="cite">
<div dir="ltr">
<div>I cannot reproduce your problem with a fairly
large amount of data - 5 GB (50 million reads) of soil
metagenomic data processed successfully with
'sandbox/filter-below-abund.py'. (I think the characteristics
of your data set are different though; I thought I noticed
some sequences with 'N' in them - those would be discarded. If
you have many of those then that could drastically reduce what
is kept which might alter the read-process-write "rhythm"
between your threads some.)</div>
<div><br>
</div>
<div>... filtering 48400000</div>
<div>done loading in sequences</div>
<div>DONE writing.</div>
<div>processed 48492066 / wrote 48441373 / removed 50693</div>
<div>processed 3940396871 bp / wrote 3915266313 bp / removed
25130558 bp</div>
<div>discarded 0.6%</div>
<div><br>
</div>
<div>When I have a fresh mind tomorrow, I will suggest some next
steps. (Try to isolate which thread is dying, build a fresh
Python 2.7 on a machine which has access to your data,
etc....)</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Mar 14, 2013 at 8:10 PM, Eric
McDonald <span dir="ltr"><<a href="mailto:emcd.msu@gmail.com" target="_blank">emcd.msu@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Alexis and <span style="font-family:arial,sans-serif;font-size:13px">Louise-Amélie,</span>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">Thank
you both for the information. I am trying to reproduce
your problem with a large data set right now.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">I
agree that the problem may be a function of the amount
of data. However, if you were running out of memory,
then I would expect to see a segmentation fault rather
than a FPE. I am still guessing this problem may be
threading-related (even if the number of workers is
reduced to 1, there is still the master thread which
supplies the groups of sequences and the writer thread
which outputs the kept sequences). But, my guesses
have not proved to be that useful with your problem
thus far, so take my latest guess with a grain of
salt. :-)</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">Depending
on whether I am able to reproduce the problem, I have
some more ideas which I intend to try tomorrow. If you
find anything else interesting, I would like to know.
But, I feel bad about how much time you have wasted on
this. Hopefully I will be able to reproduce the
problem....</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">Thanks,</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">
Eric</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
</div>
<br>
</blockquote>
</div>
</div>
</blockquote>
<br>
</div><span class="HOEnZb"><font color="#888888"><div>-- <br>
<img src="cid:part2.03000403.09050809@u-bordeaux2.fr" border="0"></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div>Eric McDonald</div><div>HPC/Cloud Software Engineer</div><div> for the Institute for Cyber-Enabled Research (iCER)</div><div> and the Laboratory for Genomics, Evolution, and Development (GED)</div>
<div>Michigan State University</div><div>P: 517-355-8733</div></div>
</div>