[bip] Reproducible research

[C. Titus Brown]

On Thu, Mar 05, 2009 at 09:51:48AM +0000, Leighton Pritchard wrote:
-> There's another issue with reproducing work from others' publications that
-> hasn't come up yet: the work is frequently described inadequately for
-> reproduction, in the methods section.
-> 
-> In my experience, this is depressingly often the case for publications that
-> apply bioinformatics.

It's true across the board; experimental protocols suffer from the same
problem, and I'm sure that in, say, chemistry, you run into the same
issue too.  There's very little incentive for an accurate description of
the process by which you arrived at your results.

I am mildly skeptical that there's significant value to demanding
exact reproducibility in many circumstances.  I can think of many
situations where it doesn't matter (finding a particular set of gene in
a genome; doing a particular statistical analysis on a data set; etc.).
Are we guilty of adhering to the dogma of reproducibility at the expense
of pragmatically cutting our losses and simply doing enough that our
steps could probably be retraced by someone who was really interested?

I don't mean this as a devil's advocate kind of argument, either.  I'm a
little uneasy with all the discussions of how to force, ensure, or
otherwise require people to publish their exact pipeline and data
transformations.  I think it's much more important that published
software be open (so I can re-run it on their or other data sets) and
published data sets be available (so I can run my own tools on their
data set) than that I can reproduce their exact results.

Of course, I also tend to be more interested in bioinformatics that
leads to testable biological hypotheses, and the ultimate goal is to get
at least some of those hypothese tested .  As I work in a very specific
area on very lowbrow bioinformatics, maybe that's not widely applicable.

cheers,
--titus
-- 
C. Titus Brown, ctb at msu.edu