[bip] Reproducible research

[James Taylor]

>> Um... no. If nobody can reproduce it, it is not science.
>
> How reproducible are astronomical observations? Are they
> science? What about those observations of rare deep-sea
> animals seen only once, which I've seen published in
> Nature?
>
> One of the theories of science is that the ability to predict,
> with falsifiability, is the key to science. Reproducibility is
> a subset of that.

Fair enough. This discussion was specifically about computational  
approaches to analyzing data. For those I believe exact  
reproducibility is necessary (but, to be clear, not sufficient for  
interestingness).

For observational data the criteria needs to be different. This  
applies to your examples (events that are so rare that even if I try I  
may never seem them again) but also to experimental data where I may  
be able to reproduce the steps, but interpretation is required to  
decide whether I have reproduced the results.

Again, taking the astronomical observations as an example, the  
observed data may be completely unique, but it also needs to undergo  
processing and filtering and statistical analysis to become  
understandable. The details of that process should be precisely  
documented and reproducible.

You are making me question this though, why is computation / analysis  
different? I experience tells us that methods and analysis details  
really do matter. Tiny parameter changes can completely alter the  
significance of results. Often the significance of certain assumptions  
isn't even known until years later when more data is collected. When  
going back and evaluating past work, I need to be 1) trust that the  
observed data isn't falsified (a combination of knowing how it was  
produced and inspecting it), and 2) understand exactly how the authors  
derived their conclusions from that data and how they assessed the  
significance of their conclusions.