[khmer] normalize-by-median.py Hanging

[C. Titus Brown]

Excellent!

---
C. Titus Brown, ctb at msu.edu

> On Sep 19, 2014, at 8:30, Daniel Standage <daniel.standage at gmail.com> wrote:
> 
> Ok sorry for the scare, it looks like it was a transient I/O issue. Second time it ran through successfully. Thanks!
> 
> 
> --
> Daniel S. Standage
> Ph.D. Candidate
> Computational Genome Science Laboratory
> Indiana University
> 
>> On Thu, Sep 18, 2014 at 9:25 AM, C. Titus Brown <ctb at msu.edu> wrote:
>> If it's not responding to kill -9, then it's an OS-level hang on I/O, most
>> likely; that's probably not khmer's fault!
>> 
>> If it responds to kill -9 but not Ctrl-C, then it's a threading issue,
>> which would not be unheard of.
>> 
>> cheers,
>> --titus
>> 
>> On Wed, Sep 17, 2014 at 04:53:09PM -0400, Daniel Standage wrote:
>> > I don't know if this helps, but: the command is not responding to Ctrl-c or
>> > kill -9 as root.
>> >
>> >
>> > --
>> > Daniel S. Standage
>> > Ph.D. Candidate
>> > Computational Genome Science Laboratory
>> > Indiana University
>> >
>> > On Wed, Sep 17, 2014 at 4:38 PM, Daniel Standage <daniel.standage at gmail.com>
>> > wrote:
>> >
>> > > Here it is: https://gist.github.com/standage/4222de94bd695f23f673
>> > >
>> > > Forgot to mention, running khmer 1.1
>> > >
>> > > Thanks,
>> > > Daniel
>> > >
>> > >
>> > > --
>> > > Daniel S. Standage
>> > > Ph.D. Candidate
>> > > Computational Genome Science Laboratory
>> > > Indiana University
>> > >
>> > > On Wed, Sep 17, 2014 at 4:31 PM, C. Titus Brown <ctb at msu.edu> wrote:
>> > >
>> > >> On Wed, Sep 17, 2014 at 04:27:09PM -0400, Daniel Standage wrote:
>> > >> > Before running norm-by-median, I
>> > >> >
>> > >> >    - Downloaded SRA file
>> > >> >    - Used fastq-dump to create paired Fastq files
>> > >> >    - used interleave-reads to create a Fastq file in the One True Format
>> > >> >
>> > >> > All of the Fastq files seem to be fine i.e. none appear truncated.
>> > >> Memory
>> > >> > usage is remaining constant, CPU utilization is 100%, but the weird
>> > >> thing
>> > >> > is that as far as I can tell the norm-by-median script is complete. It
>> > >> has
>> > >> > processed all the input, given a final report, and all of the kept reads
>> > >> > have been written to output: except the last read is missing and the
>> > >> second
>> > >> > to last read is cut off.
>> > >>
>> > >> Sounds like a classic fencepost error :(.
>> > >>
>> > >> Could you send me (on or off the list) the SRA commands you used and
>> > >> the command line you ran with diginorm?  Then we'll see if we can
>> > >> replicate
>> > >> on our own hardware.
>> > >>
>> > >> cheers,
>> > >> --titus
>> > >>
>> > >> > On Wed, Sep 17, 2014 at 4:21 PM, C. Titus Brown <ctb at msu.edu> wrote:
>> > >> >
>> > >> > > Hi Daniel,
>> > >> > >
>> > >> > > sounds like an infinite loop of some sort :(.
>> > >> > >
>> > >> > > A few questions --
>> > >> > >
>> > >> > > What version of khmer are you using?
>> > >> > >
>> > >> > > Have you run the reads file through any other software?  I'm worried
>> > >> > > that the file is truncated in some way.
>> > >> > >
>> > >> > > Do you know how far through your reads file it's gotten?
>> > >> > >
>> > >> > > Is memory usage increasing or remaining constant?
>> > >> > >
>> > >> > > thanks,
>> > >> > > --titus
>> > >> > >
>> > >> > > On Wed, Sep 17, 2014 at 04:16:37PM -0400, Daniel Standage wrote:
>> > >> > > > Hi all,
>> > >> > > >
>> > >> > > > I am seeing some strange behavior running normalize-by-median.py.
>> > >> The
>> > >> > > > program seemed to complete successfully after 30-45 minutes, but
>> > >> then it
>> > >> > > > just hung there. It's now been at least 90 minutes and it's
>> > >> continuing to
>> > >> > > > hang. The output file seems to contain all the data except the last
>> > >> > > record,
>> > >> > > > and the second-to-last record is cut off.
>> > >> > > >
>> > >> > > > (khmer-env)[standage at bggnomic qc] tail SRR494178_int.fastq.keep
>> > >> > > > +
>> > >> > > >
>> > >> > >
>> > >> GBGED>>E##################################################################################
>> > >> > > > @SRR494178.12090255/1
>> > >> > > >
>> > >> > >
>> > >> TCGAGGACNACCTTTTGACCCTTCTGCAACCTTTGAATTTCAGACATCAAACTCTCCCTCTGTCGTGTCTCCNNCAATGATGGGTCGGGC
>> > >> > > > +
>> > >> > > >
>> > >> > >
>> > >> IIIIIGGG#GGGGGGIIIIIIIIIIIIIIIIGIHIIIIIGIIIIIIIIIIIIIHIIIIHIEGHHIFIHII=?##?;9>>;IGBFFGBD8G
>> > >> > > > @SRR494178.12090255/2
>> > >> > > >
>> > >> > >
>> > >> GATTCCGTCACCGAGGAGTATCCGTTGCCGAGGTTGTGCGTCTGTCGAACCTGGCCGTTCTTTTTGACCGTGTAGGTGCCGCCGTTGATC
>> > >> > > > +
>> > >> > > > IIIIIIHIIIIIIIIIBIHHIIIGIIIIIII(khmer-env)[standage at bggnomic qc]
>> > >> > > >
>> > >> > > > Any ideas as to what could be causing this?
>> > >> > > >
>> > >> > > > Thanks,
>> > >> > > > Daniel
>> > >> > > >
>> > >> > > > PS.
>> > >> > > >
>> > >> > > >    - OS: Fedora 20 with lots o RAM (100s of GB)
>> > >> > > >    - Command: normalize-by-median.py -k 17 -p -N 4 -x 8e9
>> > >> > > >    SRR494178_int.fastq
>> > >> > > >    - Data: http://www.ncbi.nlm.nih.gov/sra/?term=SRR494178
>> > >> > > >
>> > >> > > >
>> > >> > > > --
>> > >> > > > Daniel S. Standage
>> > >> > > > Ph.D. Candidate
>> > >> > > > Computational Genome Science Laboratory
>> > >> > > > Indiana University
>> > >> > >
>> > >> > > > _______________________________________________
>> > >> > > > khmer mailing list
>> > >> > >
>> > >> > > --
>> > >> > > C. Titus Brown, ctb at msu.edu
>> > >> > >
>> > >>
>> > >> --
>> > >> C. Titus Brown, ctb at msu.edu
>> > >>
>> > >
>> > >
>> 
>> --
>> C. Titus Brown, ctb at msu.edu
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/khmer/attachments/20140919/7a42cfaa/attachment-0001.htm>