[TIP] including (or not) tests within your package

[Fernando Perez]

On Fri, Jul 30, 2010 at 5:01 AM, Olemis Lang <olemis at gmail.com> wrote:
> Probably this is a matter of flavors . The fact is that I've not seen
> yet (or probably skipped ...) a strong reason (besides my current
> laziness ;o) in favor of *always* including tests inside a package
> prepared to run in production systems .

This bug in the production, official Debian/Ubuntu delivered version of ATLAS:

https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/363510

was detected because numpy ships its test suite along with it, and someone ran

import numpy
numpy.test()

on a production system, and boom.  Yes, it could have been found
otherwise, but it was very, very useful for the numpy developers to
know this, and also to be able to ask users of other similar systems
to also run numpy.test() until we could understand where it was coming
from.

Production systems are the ones where you want to know that things
work correctly, right? So they should be the ones where you actually
test your code.

We *always* ship the tests with all projects I am involved with, and
work very hard to provide really easy ways for users to run the full
test suite and provide us with useful feedback when something goes
wrong.

I can't count the number of bugs we've found in this manner, simply
because they only appear with some combination of compiler, hardware
endianness, library versions, etc, which developers would *never* have
access to, but some user at some big government lab happens to run on.

So *my* policy is pretty simple: yes, absolutely, unconditionally,
always, ship the tests to our users, and make sure the first thing
they do before they ask for help or report problems is run the test
suite.

I've put *enormous* amounts of time and effort into making the test
suite trivial to run for users so that this policy is viable, and I'm
sure I could do better, but it's a start.

Cheers,

f