[khmer] normalize-by-median.py - IOError: [Errno 30] Read-only file system:

Michael R. Crusoe mcrusoe at msu.edu
Sun Jul 6 22:00:42 PDT 2014


A couple points:

If you're using a virtualenv you have to activate prior to other commands
and when you want to use it again from a fresh terminal. I really recommend
using pip for the install as well. Once the khmer project is installed that
way you can run the scripts just like any other command.

The '-R' option to normalize-by-median is for specifying the filename of a
report file. According to what I see in your email you specify the same
filename as your main input file. In my tests this causes the input file to
be erased, which is a bug.

The kept sequences from normalize-by-median.py are saved to a file in the
current directory with the same filename as the input but with ".keep"
appended. This is likely the source of the error you see.

Here is my suggested variation:

```
curl -O
https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.11.6.tar.gz
tar xzf virtualenv*
python2.7 virtualenv*/virtualenv.py khmerEnv
source khmerEnv/bin/activate
pip2 install khmer
cd  /work/efraim/output/
interleave-reads.py trimmed_R1_001.fasta trimmed_R2_001.fasta >
trimmed_R.fasta
normalize-by-median.py -k 20 -N 4 -x 2e9 -C 20 -p trimmed_R.fasta -R
trimmed_R.keep.report
# output will be in /work/efraim/output/trimmed_R.fasta.keep

```

Cheers,


On Mon, Jul 7, 2014 at 12:37 AM, Efraim Rodrigues <efraimnaassom at gmail.com>
wrote:

> Okay, here it goes.
>
> python interleave-reads.py /work/efraim/output/trimmed_R1_001.fasta
> /work/efraim/output/trimmed_R2_001.fasta >
> /work/efraim/khmer/output/trimmed_R.fasta
>
> python normalize-by-median.py -k 20 -N 4 -x 2e9 -C 20 -p
> /work/efraim/khmer/output/trimmed_R.fasta -R
> /work/efraim/khmer/output/normalized_R.fasta
>
> Unfortunately, I am still getting the same error. This is the output from
> the scripts interleave-reads.py and normalize-by-median.py:
>
>
> || This is the script 'interleave-reads.py' in khmer.
> || You are running khmer version 1.1
> || You are also using screed version 0.7
> ||
> || If you use this script in a publication, please cite EACH of the
> following:
> ||
> ||   * MR Crusoe et al., 2014. doi: 10.6084/m9.figshare.979190
> ||
> || Please see the CITATION file for details.
>
> Interleaving:
> /work/efraim/output/trimmed_R1_001.fasta
> /work/efraim/output/trimmed_R2_001.fasta
> ... 0 pairs
> ... 100000 pairs
> ... 200000 pairs
> ... 300000 pairs
> ... 400000 pairs
> ... 500000 pairs
> ... 600000 pairs
> ... 700000 pairs
> ... 800000 pairs
> ... 900000 pairs
> ... 1000000 pairs
> ... 1100000 pairs
> ... 1200000 pairs
> ... 1300000 pairs
> ... 1400000 pairs
> ... 1500000 pairs
> ... 1600000 pairs
> ... 1700000 pairs
> ... 1800000 pairs
> ... 1900000 pairs
> ... 2000000 pairs
> ... 2100000 pairs
> ... 2200000 pairs
> ... 2300000 pairs
> ... 2400000 pairs
> ... 2500000 pairs
> ... 2600000 pairs
> ... 2700000 pairs
> ... 2800000 pairs
> ... 2900000 pairs
> ... 3000000 pairs
> ... 3100000 pairs
> ... 3200000 pairs
> ... 3300000 pairs
> ... 3400000 pairs
> ... 3500000 pairs
> ... 3600000 pairs
> ... 3700000 pairs
> ... 3800000 pairs
> ... 3900000 pairs
> ... 4000000 pairs
> final: interleaved 4026442 pairs
>
> || This is the script 'normalize-by-median.py' in khmer.
> || You are running khmer version 1.1
> || You are also using screed version 0.7
> ||
> || If you use this script in a publication, please cite EACH of the
> following:
> ||
> ||   * MR Crusoe et al., 2014. doi: 10.6084/m9.figshare.979190
> ||   * CT Brown et al., arXiv:1203.4802 [q-bio.GN]
> ||
> || Please see the CITATION file for details.
>
>
> PARAMETERS:
>  - kmer size =    20 (-k)
>  - n tables =     4 (-N)
>  - min tablesize = 2e+09 (-x)
>
> Estimated memory usage is 8e+09 bytes (n_tables x min_tablesize)
> --------
> Traceback (most recent call last):
>   File "normalize-by-median.py", line 5, in <module>
>     pkg_resources.run_script('khmer==1.1', 'normalize-by-median.py')
>   File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 534, in
> run_script
>   File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1434, in
> run_script
>   File
> "/home/efraim/.local/lib/python2.7/site-packages/khmer-1.1-py2.7-linux-x86_64.egg/EGG-INFO/scripts/normalize-by-median.py",
> line 312, in <module>
>     main()
>   File
> "/home/efraim/.local/lib/python2.7/site-packages/khmer-1.1-py2.7-linux-x86_64.egg/EGG-INFO/scripts/normalize-by-median.py",
> line 243, in main
>     outfp = open(output_name, 'w')
> IOError: [Errno 30] Read-only file system: 'normalized_R.fasta.keep'
>
> Thank you!
>
>
>
> On Sun, Jul 6, 2014 at 11:11 PM, Michael R. Crusoe <mcrusoe at msu.edu>
> wrote:
>
>> If you could include the exact command (with parameters) that is causing
>> the problem and which directory you run it in that would be useful.
>>
>>
>> On Mon, Jul 7, 2014 at 12:09 AM, Efraim Rodrigues <
>> efraimnaassom at gmail.com> wrote:
>>
>>> Alright! Awesome!
>>>
>>> I am sorry guys, I did not see that the newest version was the 1.11.6.
>>>
>>> Well, 'make test' worked and the result was 'OK'.
>>>
>>> I'm going back to the initial problem which was error  IOError: [Errno
>>> 30] Read-only file system: . I'll try to run it again and I'll let you guys
>>> know about the result.
>>>
>>> Thanks one more time and I'm sorry about not getting the newest version.
>>>
>>>
>>> On Sun, Jul 6, 2014 at 11:01 PM, Michael R. Crusoe <mcrusoe at msu.edu>
>>> wrote:
>>>
>>>> The lack of `pip2` when installed this way is fixed in the latest
>>>> virtualenv release (1.11.6):
>>>>
>>>>  ```
>>>> curl -O
>>>> https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.11.6.tar.gz
>>>> tar xzf virtualenv*
>>>> python2.7 virtualenv*/virtualenv.py khmerEnv
>>>> source khmerEnv/bin/activate
>>>> pip2 --version
>>>> ```
>>>>
>>>>
>>>> On Sun, Jul 6, 2014 at 11:11 PM, C. Titus Brown <ctb at msu.edu> wrote:
>>>>
>>>>> On Sun, Jul 06, 2014 at 10:09:20PM -0500, Efraim Rodrigues wrote:
>>>>> > When I try to run 'make test'. It looks like it calls a script which
>>>>> runs
>>>>> > pip2. I guess I cannot change it.
>>>>> >
>>>>> > Oh, do not feel sorry. You guys are willing to help and that's all
>>>>> that
>>>>> > matters.
>>>>>
>>>>> well, thanks ;)
>>>>>
>>>>> You can do:
>>>>>
>>>>>         ln <virtualenv>/bin/pip <virtualenv>/bin/pip2
>>>>>
>>>>> where you replace <virtualenv> with the path to your virtualenv
>>>>> directory.
>>>>>
>>>>> cheers,
>>>>> --titus
>>>>>
>>>>> > On Sun, Jul 6, 2014 at 10:01 PM, C. Titus Brown <ctb at msu.edu> wrote:
>>>>> >
>>>>> > > Just try pip (not pip2). Sorry for all the mess :(
>>>>> > >
>>>>> > > On Jul 6, 2014, at 7:53 PM, Efraim Rodrigues <
>>>>> efraimnaassom at gmail.com>
>>>>> > > wrote:
>>>>> > >
>>>>> > > Hi. Thanks for your response.
>>>>> > >
>>>>> > > It looks like the we are supposed to choose the version of the
>>>>> virtualenv.
>>>>> > > I found all the versions here
>>>>> > > https://pypi.python.org/packages/source/v/virtualenv/ so I am
>>>>> followed
>>>>> > > these steps using the newest version of the virtualen, in this
>>>>> case, I
>>>>> > > believe that is,
>>>>> > >
>>>>> https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.tar.gz
>>>>> > >
>>>>> > > Anyway, I ran all the tests and it still returns the same problem.
>>>>> > >
>>>>> > >  make test
>>>>> > > pip2 install --user nose || pip2 install nose
>>>>> > > /bin/sh: pip2: command not found
>>>>> > > /bin/sh: pip2: command not found
>>>>> > > make: *** [test] Error 127
>>>>> > >
>>>>> > > In fact, there is no pip2 inside khmerEnv. I can just see these
>>>>> files:
>>>>> > >
>>>>> > > activate      activate.fish     easy_install      pip      python
>>>>> > > python2.7
>>>>> > > activate.csh  activate_this.py  easy_install-2.7  pip-2.7  python2
>>>>> > >
>>>>> > > Should I try another version of virutalenv or perhaps another
>>>>> similar tool?
>>>>> > >
>>>>> > > Thank y'all for trying to help me.
>>>>> > >
>>>>> > >
>>>>> > > On Sun, Jul 6, 2014 at 5:25 PM, C. Titus Brown <ctb at msu.edu>
>>>>> wrote:
>>>>> > >
>>>>> > >> On Sat, Jul 05, 2014 at 03:01:47PM -0500, Efraim Rodrigues wrote:
>>>>> > >> > Hi everyone,
>>>>> > >> >
>>>>> > >> > Actually, when I tried to install by running 'make test', it
>>>>> returned
>>>>> > >> the
>>>>> > >> > following message:
>>>>> > >> >
>>>>> > >> > pip2 install --user nose || pip2 install nose
>>>>> > >> > /bin/sh: pip2: command not found
>>>>> > >> > /bin/sh: pip2: command not found
>>>>> > >> >
>>>>> > >> >
>>>>> > >> > Even though we have python/2.7 installed, It looks like my
>>>>> system has
>>>>> > >> > python/2.6 as default. Hence, in order to install 'khmer', I
>>>>> followed
>>>>> > >> these
>>>>> > >> > steps:
>>>>> > >> >
>>>>> > >> > module load python/2.7
>>>>> > >> >
>>>>> > >> > python setup.py install --user
>>>>> > >> >
>>>>> > >> > Apparently, it worked to install it locally, however I'm not
>>>>> sure about
>>>>> > >> any
>>>>> > >> > additional file or module.
>>>>> > >>
>>>>> > >> OK, I would suggest running the tests inside of virtualenv.  try:
>>>>> > >>
>>>>> > >> curl -O
>>>>> > >>
>>>>> https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.x.y.tar.gz
>>>>> > >> tar xzf virtualenv*
>>>>> > >> cd virtualenv-*; python2.7 virtualenv.py ../khmerEnv; cd ..
>>>>> > >> source khmerEnv/bin/activate
>>>>> > >>
>>>>> > >> and then 'make test'.
>>>>> > >>
>>>>> > >> thanks!
>>>>> > >> --titus
>>>>> > >>
>>>>> > >>
>>>>> > >
>>>>>
>>>>> --
>>>>> C. Titus Brown, ctb at msu.edu
>>>>>
>>>>> _______________________________________________
>>>>> khmer mailing list
>>>>> khmer at lists.idyll.org
>>>>> http://lists.idyll.org/listinfo/khmer
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Michael R. Crusoe:  Programmer & Bioinformatician   mcrusoe at msu.edu
>>>>  @ the Genomics, Evolution, and Development lab; Michigan State U
>>>> http://ged.msu.edu/ http://orcid.org/0000-0002-2961-9670 @biocrusoe
>>>> <http://twitter.com/biocrusoe>
>>>>
>>>
>>>
>>
>>
>> --
>> Michael R. Crusoe:  Programmer & Bioinformatician   mcrusoe at msu.edu
>>  @ the Genomics, Evolution, and Development lab; Michigan State U
>> http://ged.msu.edu/ http://orcid.org/0000-0002-2961-9670 @biocrusoe
>> <http://twitter.com/biocrusoe>
>>
>
>


-- 
Michael R. Crusoe:  Programmer & Bioinformatician   mcrusoe at msu.edu
 @ the Genomics, Evolution, and Development lab; Michigan State U
http://ged.msu.edu/ http://orcid.org/0000-0002-2961-9670 @biocrusoe
<http://twitter.com/biocrusoe>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/khmer/attachments/20140707/c7dec831/attachment-0001.htm>


More information about the khmer mailing list