<div dir="ltr"><br><div class="gmail_extra">Hi YiJin,</div><div class="gmail_extra"><br></div><div class="gmail_extra">The partitions are different than the groups -- maybe I can help clarify the difference.</div><div class="gmail_extra">
The groups are a set of partitions which have been somewhat arbitrarily put together based on your max size argument. Basically, what happens after partitioning is that you have millions to billions of partitions and its just impractical to work with each of these individually - though it might be desired in some cases. When I want to assemble partitions, for example, its much easier to work with 100 groups of partitions than billions of partitions. To group them, the partitions are pretty much rank ordered by the number of sequences they have within them and then added to a group until the maximum size is hit.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">What happens after I group them is that I want to know the distribution of sequences within partitions, and this is where the *dist file comes in. It describes the partitions, not the groups -- I see how this is confusing and apologize. The columns in the dist file are as follows: Number of sequences, number of partitions with that number of sequences, cumulative number of partitions, cumulative number of sequences (reads)</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">To blast two partitions against each other doesn't make sense to me unless there is a reason you would think they are related (based on sequence homology). That being said, its quite possible and even likely that partitions with similar numbers of reads are related but you want to be careful with this assumption. Partitions with similar #s of reads reflect biology which has been sampled with similar coverage, which means that they *could* be related...arguably more than partitions which have very different abundances. In my hands, this seems to hold true, but I'd want to validate it a lot more before committing to anything. And it would vary by what you are actually sampling of course.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Hope this helps,</div><div class="gmail_extra">Adina</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_quote">On Mon, Jan 13, 2014 at 3:00 PM, <span dir="ltr"><<a href="mailto:khmer-request@lists.idyll.org" target="_blank">khmer-request@lists.idyll.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Send khmer mailing list submissions to<br>
<a href="mailto:khmer@lists.idyll.org">khmer@lists.idyll.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.idyll.org/listinfo/khmer" target="_blank">http://lists.idyll.org/listinfo/khmer</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:khmer-request@lists.idyll.org">khmer-request@lists.idyll.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:khmer-owner@lists.idyll.org">khmer-owner@lists.idyll.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of khmer digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Partitioning: are resulting lumps that different from<br>
each other? (YiJin Liew)<br>
2. Re: Partitioning: are resulting lumps that different from<br>
each other? (C. Titus Brown)<br>
3. Re: Partitioning: are resulting lumps that different from<br>
each other? (YiJin Liew)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Sun, 12 Jan 2014 23:28:35 +0000<br>
From: YiJin Liew <<a href="mailto:YiJin.Liew@KAUST.EDU.SA">YiJin.Liew@KAUST.EDU.SA</a>><br>
Subject: Re: [khmer] Partitioning: are resulting lumps that different<br>
from each other?<br>
To: "<a href="mailto:khmer@lists.idyll.org">khmer@lists.idyll.org</a>" <<a href="mailto:khmer@lists.idyll.org">khmer@lists.idyll.org</a>><br>
Message-ID: <<a href="mailto:52D32525.1020007@kaust.edu.sa">52D32525.1020007@kaust.edu.sa</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Thanks for the reply! Could I ask a few follow-up questions regarding<br>
the format of the .dist file then, as I can't seem to find a full<br>
description of how the file is structured?<br>
<br>
Take for example<br>
<br>
--- iowa-corn-50m.dist ---<br>
1 19750012 19750012 19750012<br>
2 2905935 22655947 25561882<br>
3 745747 23401694 27799123<br>
4 324017 23725711 29095191<br>
5 167228 23892939 29931331<br>
<snip><br>
2312 1 24356713 37268397<br>
2359 1 24356714 37270756<br>
2714 1 24356715 37273470<br>
3008 1 24356716 37276478<br>
3296530 1 24356717 40573008 <-- is this the most interesting group?<br>
<br>
I can sort of guess what the numbers mean, but let me double-check: does<br>
this indicate that there's 19.8 million clusters that are "singlets";<br>
followed by 2.9 million "doublets" etc.? Also, are columns 3 and 4<br>
cumulative figures for clusters and sequences respectively?<br>
<br>
If you don't mind, could you elaborate briefly on how groups are created<br>
based on the dist file? Judging from the line counts, I suspect that the<br>
script fills the first group with singlets till --max-size is hit, if<br>
not, continue filling with doublets, then move on to the next group once<br>
--max-size is crossed?<br>
<br>
On my data, I've tried blastn-ing the groups0000 and 0001 produced from<br>
the partitioning process, but from the results I'd wager that they're<br>
roughly the same - which was what prompted me to seek advice on how the<br>
script functioned.<br>
<br>
Apologies for the wall-of-text, thanks again for your help!<br>
<br>
Yours<br>
Yi Jin<br>
<br>
<br>
<br>
On 12/01/2014 20:51, C. Titus Brown wrote:<br>
> On Thu, Jan 09, 2014 at 02:23:29PM +0000, YiJin Liew wrote:<br>
>> Dear Dr Brown,<br>
>><br>
>> Before I delve into my sob story, I'd like to thank you (and your lab)<br>
>> for writing khmer. I must say that the digital normalisation pipeline<br>
>> proved to be an elegant method of reducing the amount of errors in<br>
>> sequencing data, and our resulting assembly have improved (and sped up<br>
>> considerably) because of your programs. Thanks.<br>
>><br>
>> After the digital normalisation pipeline, I tried out the partitioning<br>
>> pipeline as described in<br>
>> <a href="http://khmer.readthedocs.org/en/latest/partitioning-big-data.html" target="_blank">http://khmer.readthedocs.org/en/latest/partitioning-big-data.html</a>. I'm<br>
>> having some trouble wrapping my head around the results produced by<br>
>> extract-partitions.py - the resulting lumps (in group000x files) seem to<br>
>> be strongly influenced by the -X (--max-size) parameter that one uses.<br>
>><br>
>> Take for example the 1.1G Iowa corn dataset you made available online,<br>
>> specifically the<br>
><br>
> Hi YiJin,<br>
><br>
> apologies for taking so long to reply. The 'group' files output by<br>
> extract-partitions contain multiple partitions; the -X parameter controls how<br>
> many sequences, roughly, go into each group. So this is entirely expected.<br>
><br>
> Partitions are connected sequences; groups are merely collections of similarly<br>
> sized partitions.<br>
><br>
> The file to take a look at is the '.dist' file; that's the distribution<br>
> of partition sizes.<br>
><br>
> best,<br>
> --titus<br>
><br>
<br>
________________________________<br>
<br>
This message and its contents including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Sun, 12 Jan 2014 17:46:50 -0800<br>
From: "C. Titus Brown" <<a href="mailto:ctb@msu.edu">ctb@msu.edu</a>><br>
Subject: Re: [khmer] Partitioning: are resulting lumps that different<br>
from each other?<br>
To: YiJin Liew <<a href="mailto:YiJin.Liew@KAUST.EDU.SA">YiJin.Liew@KAUST.EDU.SA</a>><br>
Cc: <a href="mailto:khmer@lists.idyll.org">khmer@lists.idyll.org</a><br>
Message-ID: <<a href="mailto:20140113014650.GB30578@idyll.org">20140113014650.GB30578@idyll.org</a>><br>
Content-Type: text/plain; charset=us-ascii<br>
<br>
On Sun, Jan 12, 2014 at 11:26:50PM +0000, YiJin Liew wrote:<br>
> Thanks for the reply! Could I ask a few follow-up questions regarding<br>
> the format of the .dist file then, as I can't seem to find a full<br>
> description of how the file is structured?<br>
><br>
> take for example<br>
><br>
> --- iowa-corn-50m.dist ---<br>
> 1 19750012 19750012 19750012<br>
> 2 2905935 22655947 25561882<br>
> 3 745747 23401694 27799123<br>
> 4 324017 23725711 29095191<br>
> 5 167228 23892939 29931331<br>
> <snip><br>
> 2312 1 24356713 37268397<br>
> 2359 1 24356714 37270756<br>
> 2714 1 24356715 37273470<br>
> 3008 1 24356716 37276478<br>
> 3296530 1 24356717 40573008 <-- is this the most interesting group?<br>
><br>
> I can sort of guess what the numbers mean, but let me double-check: does<br>
> this indicate that there's 19.8 million clusters that are "singlets";<br>
> followed by 2.9 million "doublets" etc.? Also, are columns 3 and 4<br>
> cumulative figures for clusters and sequences respectively?<br>
<br>
Exactly!<br>
<br>
> If you don't mind, could you elaborate briefly on how groups are created<br>
> based on the dist file? Judging from the line counts, I suspect that the<br>
> script fills the first group with singlets till --max-size is hit, if<br>
> not, continue filling with doublets, then move on to the next group once<br>
> --max-size is crossed?<br>
<br>
Yep.<br>
<br>
> On my data, I've tried blastn-ing the groups0000 and 0001 produced from<br>
> the partitioning process, but from the results I'd wager that they're<br>
> roughly the same - which was what prompted me to seek advice on how the<br>
> script functioned.<br>
<br>
Roughly the same... no, shouldn't be. Those are probably spurious<br>
BLAST matches of some sort. If partitioning worked (and at least from<br>
the examples above you got a lot of partitions) then those reads are<br>
from different components of the overall de Bruijn graph.<br>
<br>
cheers,<br>
--titus<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Mon, 13 Jan 2014 14:21:30 +0000<br>
From: YiJin Liew <<a href="mailto:YiJin.Liew@KAUST.EDU.SA">YiJin.Liew@KAUST.EDU.SA</a>><br>
Subject: Re: [khmer] Partitioning: are resulting lumps that different<br>
from each other?<br>
To: "<a href="mailto:titus@idyll.org">titus@idyll.org</a>" <<a href="mailto:titus@idyll.org">titus@idyll.org</a>><br>
Cc: "<a href="mailto:khmer@lists.idyll.org">khmer@lists.idyll.org</a>" <<a href="mailto:khmer@lists.idyll.org">khmer@lists.idyll.org</a>><br>
Message-ID: <<a href="mailto:52D3F66B.6030200@kaust.edu.sa">52D3F66B.6030200@kaust.edu.sa</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Thanks for the confirmations!<br>
<br>
To back my intuition, I repeated the rough blastn searches on<br>
groups0000, 0001 and 0002 produced from your Iowa corn dataset.<br>
<br>
What I did was:<br>
1. blastn the 1 mil reads of each dataset against nt<br>
2. for each read, pick the top hit with e-value of less than 1e-5<br>
3. and do a bit of BASH-fu to record what species the read comes from,<br>
and show the top 20.<br>
<br>
Results!<br>
<br>
--- iowa-corn-50m.group0000_vs_nt.blastn.tsv ---<br>
3173 Rhodanobacter sp.<br>
2122 Rhodopseudomonas palustris<br>
1971 Bradyrhizobium japonicum<br>
1878 Streptomyces sp.<br>
1822 Bradyrhizobium sp.<br>
1202 Ramlibacter tataouinensis<br>
1024 Variovorax paradoxus<br>
890 Streptomyces fulvissimus<br>
801 Uncultured bacterium<br>
774 Intrasporangium calvum<br>
687 Kribbella flavida<br>
616 Agromonas oligotrophica<br>
568 Actinoplanes sp.<br>
539 Conexibacter woesei<br>
518 Streptomyces griseus<br>
508 Nocardioides sp.<br>
502 Streptosporangium roseum<br>
466 Micromonospora sp.<br>
432 Verrucosispora maris<br>
390 Clavibacter michiganensis<br>
<br>
--- iowa-corn-50m.group0001_vs_nt.blastn.tsv ---<br>
5072 Rhodanobacter sp.<br>
3322 Streptomyces sp.<br>
2287 Bradyrhizobium japonicum<br>
2164 Rhodopseudomonas palustris<br>
2129 Bradyrhizobium sp.<br>
1658 Ramlibacter tataouinensis<br>
1486 Streptomyces fulvissimus<br>
1325 Intrasporangium calvum<br>
1271 Variovorax paradoxus<br>
889 Uncultured bacterium<br>
841 Streptomyces griseus<br>
728 Agromonas oligotrophica<br>
627 Nocardioides sp.<br>
626 Actinoplanes sp.<br>
555 Streptosporangium roseum<br>
552 Kribbella flavida<br>
546 Conexibacter woesei<br>
517 Micromonospora sp.<br>
474 Verrucosispora maris<br>
434 Nitrobacter hamburgensis<br>
<br>
--- iowa-corn-50m.group0002_vs_nt.blastn.tsv ---<br>
7099 Rhodanobacter sp.<br>
4219 Streptomyces sp.<br>
2559 Bradyrhizobium japonicum<br>
2399 Bradyrhizobium sp.<br>
2336 Rhodopseudomonas palustris<br>
1931 Intrasporangium calvum<br>
1852 Ramlibacter tataouinensis<br>
1803 Streptomyces fulvissimus<br>
1350 Variovorax paradoxus<br>
1170 Streptomyces griseus<br>
849 Agromonas oligotrophica<br>
848 Uncultured bacterium<br>
668 Conexibacter woesei<br>
646 Actinoplanes sp.<br>
615 Streptosporangium roseum<br>
585 Nocardioides sp.<br>
559 Micromonospora sp.<br>
491 Clavibacter michiganensis<br>
473 Kribbella flavida<br>
454 Rubrivivax gelatinosus<br>
<br>
<br>
... come to think of it, I should've repeated the same analysis on the<br>
biggest lump (group0007). However, from what I've observed from my own<br>
datasets, the biggest lump should be different from the smaller lumps.<br>
It's just that I feel that the small lumps are actually not very<br>
different from each other, and they're split mainly because of the -X<br>
setting. Hope this (very rough) analysis of mine illustrates what I'm<br>
trying to say!<br>
<br>
Yours<br>
Yi Jin<br>
<br>
On 13/01/2014 04:46, C. Titus Brown wrote:<br>
> On Sun, Jan 12, 2014 at 11:26:50PM +0000, YiJin Liew wrote:<br>
>> Thanks for the reply! Could I ask a few follow-up questions regarding<br>
>> the format of the .dist file then, as I can't seem to find a full<br>
>> description of how the file is structured?<br>
>><br>
>> take for example<br>
>><br>
>> --- iowa-corn-50m.dist ---<br>
>> 1 19750012 19750012 19750012<br>
>> 2 2905935 22655947 25561882<br>
>> 3 745747 23401694 27799123<br>
>> 4 324017 23725711 29095191<br>
>> 5 167228 23892939 29931331<br>
>> <snip><br>
>> 2312 1 24356713 37268397<br>
>> 2359 1 24356714 37270756<br>
>> 2714 1 24356715 37273470<br>
>> 3008 1 24356716 37276478<br>
>> 3296530 1 24356717 40573008 <-- is this the most interesting group?<br>
>><br>
>> I can sort of guess what the numbers mean, but let me double-check: does<br>
>> this indicate that there's 19.8 million clusters that are "singlets";<br>
>> followed by 2.9 million "doublets" etc.? Also, are columns 3 and 4<br>
>> cumulative figures for clusters and sequences respectively?<br>
><br>
> Exactly!<br>
><br>
>> If you don't mind, could you elaborate briefly on how groups are created<br>
>> based on the dist file? Judging from the line counts, I suspect that the<br>
>> script fills the first group with singlets till --max-size is hit, if<br>
>> not, continue filling with doublets, then move on to the next group once<br>
>> --max-size is crossed?<br>
><br>
> Yep.<br>
><br>
>> On my data, I've tried blastn-ing the groups0000 and 0001 produced from<br>
>> the partitioning process, but from the results I'd wager that they're<br>
>> roughly the same - which was what prompted me to seek advice on how the<br>
>> script functioned.<br>
><br>
> Roughly the same... no, shouldn't be. Those are probably spurious<br>
> BLAST matches of some sort. If partitioning worked (and at least from<br>
> the examples above you got a lot of partitions) then those reads are<br>
> from different components of the overall de Bruijn graph.<br>
><br>
> cheers,<br>
> --titus<br>
><br>
<br>
________________________________<br>
<br>
This message and its contents including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
khmer mailing list<br>
<a href="mailto:khmer@lists.idyll.org">khmer@lists.idyll.org</a><br>
<a href="http://lists.idyll.org/listinfo/khmer" target="_blank">http://lists.idyll.org/listinfo/khmer</a><br>
<br>
<br>
End of khmer Digest, Vol 12, Issue 7<br>
************************************<br>
</blockquote></div><br></div></div>