<div dir="ltr">The new version is in a complete new directory. make test gives:<div><br></div><div><div>make test</div><div>cd lib && \</div><div>make</div><div>make[1]: Entering directory `/u2/home/rajatroy/khmer/lib'</div>
<div>make[1]: Nothing to be done for `all'.</div><div>make[1]: Leaving directory `/u2/home/rajatroy/khmer/lib'</div><div>cd python && \</div><div>make DEFINE_KHMER_EXTRA_SANITY_CHECKS="" \</div>
<div> CXX_DEBUG_FLAGS="" </div><div>make[1]: Entering directory `/u2/home/rajatroy/khmer/python'</div><div>python setup.py build_ext -i</div><div>running build_ext</div><div>copying build/lib.linux-x86_64-2.7/khmer/_khmermodule.so -> khmer</div>
<div>make[1]: Leaving directory `/u2/home/rajatroy/khmer/python'</div><div>nosetests -v -x -a \!known_failing</div><div>tests.test_align.test_alignnocov ... ok</div><div>tests.test_align.test_readalign ... ok</div><div>
tests.test_align.test_alignerrorregion ... ok</div><div>tests.test_c_wrapper.test_raise_in_consume_fasta ... ok</div><div>tests.test_c_wrapper.test_raise_in_fasta_file_to_minmax ... ok</div><div>tests.test_counting_hash.Test_CountingHash.test_collision_1 ... ok</div>
<div>tests.test_counting_hash.Test_CountingHash.test_collision_2 ... ok</div><div>tests.test_counting_hash.Test_CountingHash.test_collision_3 ... ok</div><div>tests.test_counting_hash.test_3_tables ... ok</div><div>tests.test_counting_hash.test_simple_median ... ok</div>
<div>tests.test_counting_hash.test_simple_kadian ... ok</div><div>tests.test_counting_hash.test_simple_kadian_2 ... ok</div><div>tests.test_counting_hash.test_2_kadian ... ok</div><div>tests.test_counting_hash.test_save_load ... ok</div>
<div>tests.test_counting_hash.test_load_gz ... ok</div><div>tests.test_counting_hash.test_save_load_gz ... ok</div><div>tests.test_counting_hash.test_trim_full ... ok</div><div>tests.test_counting_hash.test_trim_short ... ok</div>
<div>tests.test_counting_hash.test_maxcount ... ok</div><div>tests.test_counting_hash.test_maxcount_with_bigcount ... ok</div><div>tests.test_counting_hash.test_maxcount_with_bigcount_save ... ok</div><div>tests.test_counting_hash.test_bigcount_save ... ok</div>
<div>tests.test_counting_hash.test_nobigcount_save ... ok</div><div>tests.test_counting_hash.test_bigcount_abund_dist ... ok</div><div>tests.test_counting_hash.test_bigcount_abund_dist_2 ... ok</div><div>tests.test_counting_hash.test_bigcount_overflow ... ok</div>
<div>tests.test_counting_hash.test_get_ksize ... ok</div><div>tests.test_counting_hash.test_get_hashsizes ... ok</div><div>tests.test_counting_single.Test_AbundanceDistribution.test_count_A ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_abundance_by_pos ... ok</div>
<div>tests.test_counting_single.Test_ConsumeString.test_abundance_by_pos_bigcount ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_bounded ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_bounded_2 ... ok</div>
<div>tests.test_counting_single.Test_ConsumeString.test_bounded_2_rc ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_bounded_rc ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_max_count ... ok</div>
<div>tests.test_counting_single.Test_ConsumeString.test_max_count_in_bound ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_max_count_out_bound ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_min_count ... ok</div>
<div>tests.test_counting_single.Test_ConsumeString.test_min_count_in_bound ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_min_count_out_bound ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_n_occupied ... ok</div>
<div>tests.test_counting_single.Test_ConsumeString.test_n_occupied_args ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_simple ... ok</div><div>tests.test_counting_single.Test_ConsumeString.test_simple_2 ... ok</div>
<div>tests.test_counting_single.Test_ConsumeString.test_simple_rc ... ok</div><div>tests.test_counting_single.test_no_collision ... ok</div><div>tests.test_counting_single.test_collision ... ok</div><div>tests.test_counting_single.test_complete_no_collision ... ok</div>
<div>tests.test_counting_single.test_complete_2_collision ... ok</div><div>tests.test_counting_single.test_complete_4_collision ... ok</div><div>tests.test_counting_single.test_maxcount ... ok</div><div>tests.test_counting_single.test_maxcount_with_bigcount ... ok</div>
<div>tests.test_counting_single.test_consume_uniqify_first ... ok</div><div>tests.test_counting_single.test_maxcount_consume ... ok</div><div>tests.test_counting_single.test_maxcount_consume_with_bigcount ... ok</div><div>
tests.test_counting_single.test_get_mincount ... ok</div><div>tests.test_counting_single.test_get_maxcount ... ok</div><div>tests.test_counting_single.test_get_maxcount_rc ... ok</div><div>tests.test_counting_single.test_get_mincount_rc ... ok</div>
<div>tests.test_counting_single.test_64bitshift ... ok</div><div>tests.test_counting_single.test_64bitshift_2 ... ok</div><div>tests.test_counting_single.test_very_short_read ... ok</div><div>tests.test_filter.Test_Filter.test_abund ... ok</div>
<div>tests.test_filter.test_filter_sodd ... ok</div><div>tests.test_functions.test_forward_hash ... ok</div><div>tests.test_functions.test_forward_hash_no_rc ... ok</div><div>tests.test_functions.test_reverse_hash ... ok</div>
<div>tests.test_functions.test_get_primes ... ok</div><div>tests.test_graph.Test_ExactGraphFu.test_counts ... ok</div><div>tests.test_graph.Test_ExactGraphFu.test_graph_links_next_a ... ok</div><div>tests.test_graph.Test_ExactGraphFu.test_graph_links_next_c ... ok</div>
<div>tests.test_graph.Test_ExactGraphFu.test_graph_links_next_g ... ok</div><div>tests.test_graph.Test_ExactGraphFu.test_graph_links_next_t ... ok</div><div>tests.test_graph.Test_ExactGraphFu.test_graph_links_prev_a ... ok</div>
<div>tests.test_graph.Test_ExactGraphFu.test_graph_links_prev_c ... ok</div><div>tests.test_graph.Test_ExactGraphFu.test_graph_links_prev_g ... ok</div><div>tests.test_graph.Test_ExactGraphFu.test_graph_links_prev_t ... ok</div>
<div>tests.test_graph.Test_InexactGraphFu.test_graph_links_next_a ... ok</div><div>tests.test_graph.Test_InexactGraphFu.test_graph_links_next_c ... ok</div><div>tests.test_graph.Test_InexactGraphFu.test_graph_links_next_g ... ok</div>
<div>tests.test_graph.Test_InexactGraphFu.test_graph_links_next_t ... ok</div><div>tests.test_graph.Test_InexactGraphFu.test_graph_links_prev_a ... ok</div><div>tests.test_graph.Test_InexactGraphFu.test_graph_links_prev_c ... ok</div>
<div>tests.test_graph.Test_InexactGraphFu.test_graph_links_prev_g ... ok</div><div>tests.test_graph.Test_InexactGraphFu.test_graph_links_prev_t ... ok</div><div>tests.test_graph.Test_Partitioning.test_connected_20_a ... ok</div>
<div>tests.test_graph.Test_Partitioning.test_connected_20_b ... ok</div><div>tests.test_graph.Test_Partitioning.test_connected_31_c ... ok</div><div>tests.test_graph.Test_Partitioning.test_disconnected_20_a ... ok</div><div>
tests.test_graph.Test_Partitioning.test_disconnected_20_b ... ok</div><div>tests.test_graph.Test_Partitioning.test_disconnected_31_c ... ok</div><div>tests.test_graph.Test_Partitioning.test_not_output_unassigned ... ok</div>
<div>tests.test_graph.Test_Partitioning.test_output_unassigned ... ok</div><div>tests.test_graph.Test_PythonAPI.test_ordered_connect ... ok</div><div>tests.test_hashbits.test__get_set_tag_density ... ok</div><div>tests.test_hashbits.test_n_occupied_1 ... ok</div>
<div>tests.test_hashbits.test_bloom_python_1 ... ok</div><div>tests.test_hashbits.test_bloom_c_1 ... ok</div><div>tests.test_hashbits.test_n_occupied_2 ... ok</div><div>tests.test_hashbits.test_bloom_c_2 ... ok</div><div>
tests.test_hashbits.test_filter_if_present ... ok</div><div>tests.test_hashbits.test_combine_pe ... ok</div><div>tests.test_hashbits.test_load_partitioned ... ok</div><div>tests.test_hashbits.test_count_within_radius_simple ... ok</div>
<div>tests.test_hashbits.test_count_within_radius_big ... ok</div><div>tests.test_hashbits.test_count_kmer_degree ... ok</div><div>tests.test_hashbits.test_find_radius_for_volume ... ok</div><div>tests.test_hashbits.test_circumference ... ok</div>
<div>tests.test_hashbits.test_save_load_tagset ... ok</div><div>tests.test_hashbits.test_save_load_tagset_noclear ... ok</div><div>tests.test_hashbits.test_stop_traverse ... ok</div><div>tests.test_hashbits.test_tag_across_stoptraverse ... ok</div>
<div>tests.test_hashbits.test_notag_across_stoptraverse ... ok</div><div>tests.test_hashbits.test_find_stoptags ... ok</div><div>tests.test_hashbits.test_find_stoptags2 ... ok</div><div>tests.test_hashbits.test_get_ksize ... ok</div>
<div>tests.test_hashbits.test_get_hashsizes ... ok</div><div>tests.test_hashbits.test_extract_unique_paths_0 ... ok</div><div>tests.test_hashbits.test_extract_unique_paths_1 ... ok</div><div>tests.test_hashbits.test_extract_unique_paths_2 ... ok</div>
<div>tests.test_hashbits.test_extract_unique_paths_3 ... ok</div><div>tests.test_hashbits.test_extract_unique_paths_4 ... ok</div><div>tests.test_hashbits.test_find_unpart ... ok</div><div>tests.test_hashbits.test_find_unpart_notraverse ... ok</div>
<div>tests.test_hashbits.test_find_unpart_fail ... ok</div><div>tests.test_hashbits.test_simple_median ... ok</div><div>Verify that 'has_extra_sanity_checks' exists. ... ok</div><div>Verify that all of the various attributes exist. ... ok</div>
<div>Verify that all of the various attributes exist. ... ok</div><div>Verify that all of the various attributes exist. ... ok</div><div>Verify that all of the various attributes exist. ... ok</div><div>Verify that the number of threads set is what is reported. ... ok</div>
<div>Verify that the reads file chunk size is what is reported. ... ok</div><div>tests.test_ktable.Test_KTable.test_basic ... ok</div><div>tests.test_ktable.Test_KTable.test_clear ... ok</div><div>tests.test_ktable.Test_KTable.test_consume ... ok</div>
<div>tests.test_ktable.Test_KTable.test_hash ... ok</div><div>tests.test_ktable.Test_KTable.test_intersection ... ok</div><div>tests.test_ktable.Test_KTable.test_operator_in ... ok</div><div>tests.test_ktable.Test_KTable.test_populate ... ok</div>
<div>tests.test_ktable.Test_KTable.test_update ... ok</div><div>tests.test_ktable.test_rc ... ok</div><div>tests.test_ktable.test_KmerCount ... ok</div><div>tests.test_lump.test_fakelump_together ... ok</div><div>tests.test_lump.test_fakelump_stop ... ok</div>
<div>tests.test_lump.test_fakelump_stop2 ... ok</div><div>tests.test_lump.test_fakelump_repartitioning ... ok</div><div>tests.test_minmax.Test_Basic.test_max_1 ... ok</div><div>tests.test_minmax.Test_Basic.test_max_2 ... ok</div>
<div>tests.test_minmax.Test_Basic.test_merge_1 ... ok</div><div>tests.test_minmax.Test_Basic.test_merge_2 ... ok</div><div>tests.test_minmax.Test_Basic.test_merge_3 ... ok</div><div>tests.test_minmax.Test_Basic.test_merge_4 ... ok</div>
<div>tests.test_minmax.Test_Basic.test_min_1 ... ok</div><div>tests.test_minmax.Test_Basic.test_min_2 ... ok</div><div>tests.test_minmax.Test_Basic.test_tablesize ... ok</div><div>tests.test_minmax.Test_Filestuff.test_save_no_load ... ok</div>
<div>tests.test_minmax.Test_Filestuff.test_saveload ... ok</div><div>tests.test_read_parsers.test_read_properties ... ok</div><div>tests.test_read_parsers.test_with_default_arguments ... ok</div><div>tests.test_read_parsers.test_gzip_decompression ... ok</div>
<div>tests.test_read_parsers.test_bzip2_decompression ... ok</div><div>tests.test_read_parsers.test_with_multiple_threads ... ok</div><div>tests.test_read_parsers.test_old_illumina_pair_mating ... ok</div><div>tests.test_read_parsers.test_casava_1_8_pair_mating ... ok</div>
<div>tests.test_read_parsers.test_iterator_identities ... ok</div><div>tests.test_read_parsers.test_read_pair_iterator_in_error_mode_xfail ... ok</div><div>tests.test_scripts.test_load_into_counting ... ok</div><div>tests.test_scripts.test_load_into_counting_fail ... ok</div>
<div>tests.test_scripts.test_filter_abund_1 ... ok</div><div>tests.test_scripts.test_filter_abund_2 ... ok</div><div>tests.test_scripts.test_filter_abund_3_fq_retained ... ok</div><div>tests.test_scripts.test_filter_abund_1_singlefile ... ok</div>
<div>tests.test_scripts.test_filter_abund_4_retain_low_abund ... ok</div><div>tests.test_scripts.test_filter_abund_5_trim_high_abund ... ok</div><div>tests.test_scripts.test_filter_abund_6_trim_high_abund_Z ... ok</div><div>
tests.test_scripts.test_filter_stoptags ... ok</div><div>tests.test_scripts.test_normalize_by_median ... ok</div><div>tests.test_scripts.test_normalize_by_median_2 ... ok</div><div>tests.test_scripts.test_normalize_by_median_paired ... ok</div>
<div>tests.test_scripts.test_normalize_by_median_impaired ... ok</div><div>tests.test_scripts.test_normalize_by_median_force ... ok</div><div>tests.test_scripts.test_normalize_by_median_dumpfrequency ... ok</div><div>tests.test_scripts.test_normalize_by_median_empty ... ok</div>
<div>tests.test_scripts.test_count_median ... ok</div><div>tests.test_scripts.test_load_graph ... ok</div><div>tests.test_scripts.test_load_graph_no_tags ... ok</div><div>tests.test_scripts.test_load_graph_fail ... ok</div>
<div>tests.test_scripts.test_partition_graph_1 ... ok</div><div>tests.test_scripts.test_partition_graph_nojoin_k21 ... ok</div><div>tests.test_scripts.test_partition_graph_nojoin_stoptags ... ok</div><div>tests.test_scripts.test_partition_graph_big_traverse ... ok</div>
<div>tests.test_scripts.test_partition_graph_no_big_traverse ... ok</div><div>tests.test_scripts.test_annotate_partitions ... ok</div><div>tests.test_scripts.test_annotate_partitions_2 ... ok</div><div>tests.test_scripts.test_extract_partitions ... ok</div>
<div>tests.test_scripts.test_abundance_dist ... ok</div><div>tests.test_scripts.test_abundance_dist_nobigcount ... ok</div><div>tests.test_scripts.test_abundance_dist_single ... ok</div><div>tests.test_scripts.test_abundance_dist_single_nobigcount ... ok</div>
<div>tests.test_scripts.test_do_partition ... ok</div><div>tests.test_scripts.test_do_partition_2 ... ok</div><div>tests.test_scripts.test_interleave_reads_1_fq ... ok</div><div>tests.test_scripts.test_interleave_reads_2_fa ... ok</div>
<div>tests.test_scripts.test_extract_paired_reads_1_fa ... ok</div><div>tests.test_scripts.test_extract_paired_reads_2_fq ... ok</div><div>tests.test_scripts.test_split_paired_reads_1_fa ... ok</div><div>tests.test_scripts.test_split_paired_reads_2_fq ... ok</div>
<div>tests.test_split.test_2_split ... ok</div><div>tests.test_split.test_n_split ... ok</div><div>tests.test_split.test_n3_split ... ok</div><div>tests.test_subset_graph.Test_RandomData.test_3_merge_013 ... ok</div><div>
tests.test_subset_graph.Test_RandomData.test_3_merge_023 ... ok</div><div>tests.test_subset_graph.Test_RandomData.test_5_merge_046 ... ok</div><div>tests.test_subset_graph.Test_RandomData.test_random_20_a_succ ... ok</div>
<div>tests.test_subset_graph.Test_RandomData.test_random_20_a_succ_II ... ok</div><div>tests.test_subset_graph.Test_RandomData.test_random_20_a_succ_III ... ok</div><div>tests.test_subset_graph.Test_RandomData.test_random_20_a_succ_IV ... ok</div>
<div>tests.test_subset_graph.Test_RandomData.test_random_20_a_succ_IV_save ... ok</div><div>tests.test_subset_graph.Test_SaveLoadPmap.test_save_load_merge ... ok</div><div>tests.test_subset_graph.Test_SaveLoadPmap.test_save_load_merge_2 ... ok</div>
<div>tests.test_subset_graph.Test_SaveLoadPmap.test_save_merge_from_disk ... ok</div><div>tests.test_subset_graph.Test_SaveLoadPmap.test_save_merge_from_disk_2 ... ok</div><div>tests.test_subset_graph.test_output_partitions ... ok</div>
<div>tests.test_subset_graph.test_tiny_real_partitions ... ok</div><div>tests.test_subset_graph.test_small_real_partitions ... ok</div><div>tests.test_threaded_sequence_processor.test_basic ... ok</div><div>tests.test_threaded_sequence_processor.test_basic_fastq_like ... ok</div>
<div>tests.test_threaded_sequence_processor.test_odd ... ok</div><div>tests.test_threaded_sequence_processor.test_basic_2thread ... ok</div><div>tests.test_threaded_sequence_processor.test_paired_2thread ... ok</div><div>
tests.test_threaded_sequence_processor.test_paired_2thread_more_seq ... ok</div><div><br></div><div>----------------------------------------------------------------------</div><div>Ran 233 tests in 20.632s</div><div><br></div>
<div>OK</div></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 27, 2013 at 5:29 PM, C. Titus Brown <span dir="ltr"><<a href="mailto:ctb@msu.edu" target="_blank">ctb@msu.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>Hmm, make sure you've deleted old versions of Khmer. What does 'make test' report in the top Khmer directory?<br>
<br><div>---</div>C. Titus Brown, <a href="mailto:ctb@msu.edu" target="_blank">ctb@msu.edu</a></div><div><div class="h5"><div><br>On Aug 27, 2013, at 17:27, Rajat Shuvro Roy <<a href="mailto:rajatroy@cs.rutgers.edu" target="_blank">rajatroy@cs.rutgers.edu</a>> wrote:<br>
<br></div><blockquote type="cite"><div><div dir="ltr">Thanks so much. I downloaded and compiled the latest version. make test resulted in 'ok' for everything. However, when I tried to run it, I get the following message:<div>
<br></div><div><div>python load-into-counting.py -k 31 -x 5e10 <a href="http://out.kh" target="_blank">out.kh</a> 1Mreads.fa </div>
<div>Traceback (most recent call last):</div><div> File "load-into-counting.py", line 13, in <module></div><div> from khmer.counting_args import build_construct_args, report_on_config</div><div>ImportError: cannot import name report_on_config</div>
</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 27, 2013 at 4:41 PM, C. Titus Brown <span dir="ltr"><<a href="mailto:ctb@msu.edu" target="_blank">ctb@msu.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Rajat,<br>
<br>
sorry for long delay in response!<br>
<div><br>
On Thu, Jul 18, 2013 at 03:32:39PM -0400, Rajat Shuvro Roy wrote:<br>
</div><div><div>> Hello Prof Brown,<br>
> I was attempting to produce a k-mer frequency distribution using khmer and<br>
> followed the instructions in (<br>
> <a href="http://khmer.readthedocs.org/en/latest/scripts.html" target="_blank">http://khmer.readthedocs.org/en/latest/scripts.html</a>) . I have a Zia mays<br>
> library (SRR404240, 95.8Gbp ) and I executed the following command.<br>
><br>
> python load-into-counting.py -k 31 -x 5e10 <a href="http://out.kh" target="_blank">out.kh</a> SRR404240.fasta<br>
><br>
> I believe, this counts k-mer frequencies and the script abundance-dist.py<br>
> produces the distribution.<br>
><br>
> We stopped it after it had ran for 2464 mins (41hrs) using 187GB space. I<br>
> tried with smaller values for -x but failed to complete the computation in<br>
> less than 3 days. Could you please let us know if this is expected and we<br>
> should allow more time. And is there a more efficient way of using Khmer?<br>
<br>
</div></div>Your e-mail actually triggered some doc changes and updates ;).<br>
<br>
Briefly, khmer can count k-mers in either constant-memory mode or in<br>
accurate-large-counts mode. In the former, counts above 255 will<br>
stop being counted, but the memory specified with the -N and -x parameters<br>
will be the total amount used; in the latter mode (which is the default),<br>
counts above 255 will be kept and memory use will expand indefinitely.<br>
<br>
You can use these modes easily in the latest khmer, the bleeding-edge<br>
branch; you can get that like so:<br>
<br>
git clone <a href="https://github.com/ged-lab/khmer.git" target="_blank">https://github.com/ged-lab/khmer.git</a> -b bleeding-edge<br>
<br>
Then use 'load-into-counting.py -b' to build the tables, and 'abundance-dist'<br>
to generate the output.<br>
<br>
I'd suggest running it on a small test data set (data/25k.fq.gz, in the<br>
khmer repo) just to make sure it all works for you, but it should - we use<br>
this regularly.<br>
<br>
Please let me know if you have any questions, and again, apologies for<br>
the delay!<br>
<br>
cheers,<br>
--titus<br>
<div><div>--<br>
C. Titus Brown, <a href="mailto:ctb@msu.edu" target="_blank">ctb@msu.edu</a><br>
</div></div></blockquote></div><br></div>
</div></blockquote></div></div></div></blockquote></div><br></div>