[bip] Reproducible research

[Leighton Pritchard]

On 05/03/2009 09:25, "Ivan Rossi" <ivan at biodec.com> wrote:

> On Wed, 4 Mar 2009, C. Titus Brown wrote:

>> Unfortunately doing both good code + good publication is an awful lot of
>> hard work, so if it's not rewarded *shrug*.
> 
> Amen, Brother Titus.
> 
> That's why some famous an not-so famous journals are so full of sh^H^H
> results-on-well-tempered-data-sets to be close to useless many many times.

There's another issue with reproducing work from others' publications that
hasn't come up yet: the work is frequently described inadequately for
reproduction, in the methods section.

In my experience, this is depressingly often the case for publications that
apply bioinformatics.  For example, many methods sections will describe,
vaguely, the use of a BLAST search without specifying enough details
(substitution matrix, gap opening/extension parameters, etc.) even to repeat
the search.  This problem extends to the exclusion of (sometimes multiple,
and sometimes egregiously) essential steps, reporting results based on
unreleased datasets or software, and even in some cases the outright absence
of any methodological information.  One particularly maddening case I read
involved undocumented manual additions to a training set halfway through an
'automated' search.  I'm sure we've all got a long list of such examples we
could cite.

I suspect that this is, at least in part, because many reviewers either
don't bother to read that section or, when they do, they don't understand
what needs to be presented to make such an analysis reproducible.  This is
particularly likely when no bioinformatician is consulted or used as a
reviewer.  Biologically-inclined authors may be urged to compress methods
sections as far as possible to squeeze in more data/discussion, and may not
be sympathetic to, or understand, the need for a methodological description
of the bioinformatic work that is reproducible, when writing the article.

It's amazing that Science works at all, sometimes ;)

L.

-- 
Dr Leighton Pritchard MRSC
D131, Plant Pathology Programme, SCRI
Errol Road, Invergowrie, Perth and Kinross, Scotland, DD2 5DA
e:lpritc at scri.ac.uk       w:http://www.scri.ac.uk/staff/leightonpritchard
gpg/pgp: 0xFEFC205C       tel:+44(0)1382 562731 x2405


______________________________________________________________________
SCRI, Invergowrie, Dundee, DD2 5DA.  
The Scottish Crop Research Institute is a charitable company limited by
guarantee. 
Registered in Scotland No: SC 29367.
Recognised by the Inland Revenue as a Scottish Charity No: SC 006662.


DISCLAIMER:

This email is from the Scottish Crop Research Institute, but the views 
expressed by the sender are not necessarily the views of SCRI and its 
subsidiaries.  This email and any files transmitted with it are
confidential

to the intended recipient at the e-mail address to which it has been 
addressed.  It may not be disclosed or used by any other than that
addressee.
If you are not the intended recipient you are requested to preserve this

confidentiality and you must not use, disclose, copy, print or rely on
this 
e-mail in any way. Please notify postmaster at scri.ac.uk quoting the 
name of the sender and delete the email from your system.

Although SCRI has taken reasonable precautions to ensure no viruses are 
present in this email, neither the Institute nor the sender accepts any 
responsibility for any viruses, and it is your responsibility to scan
the email and the attachments (if any).
______________________________________________________________________