[TIP] including (or not) tests within your package

[Fernando Perez]

On Wed, Aug 11, 2010 at 12:38 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Fernando Perez <fperez.net at gmail.com> writes:
>
>> We *always* ship the tests with all projects I am involved with, and
>> work very hard to provide really easy ways for users to run the full
>> test suite and provide us with useful feedback when something goes
>> wrong.
>
> I assume this means your automated tests don't have dependencies
> additional to that of the non-test parts of your code.
>
> That's a pity, because it means a lot of good testing frameworks must be
> ignored, or re-implemented, or (worst of all!) installed along with your
> code as another potentially-divergent copy of the same framework.

No, we do have nose as a dependency (for the tests only).  Since most
of the projects I'm involved with have dependencies far more complex
than nose, this has never been an issue.  IPython, numpy, scipy, the
nipy projects, all use nose and add a thin layer on top of
project-specific support (assertions for arrays, ipython special
syntax support, etc).

>> So *my* policy is pretty simple: yes, absolutely, unconditionally,
>> always, ship the tests to our users, and make sure the first thing
>> they do before they ask for help or report problems is run the test
>> suite.
>
> This would involve getting users to install extra dependencies (e.g.
> ‘nose’, ‘setuptools’, ‘test-scenarios’, etc.) for most of my test stuff,
> which makes it rather onerous to expect that to be the first thing users
> do on encountering a problem.

Just nose.  And obviously we do help users who have questions without
the test suite, but often running the tests is a very easy way for
users to find something wrong.  And we do get *tons* of users
reporting problems found by the test suite, well before they run into
an issue themselves.  Having the test suite 'just there' lets it act
as an 'early warning system' for bugs.

>> I've put *enormous* amounts of time and effort into making the test
>> suite trivial to run for users so that this policy is viable, and I'm
>> sure I could do better, but it's a start.
>
> On the other hand, I've made use of the good testing and development
> tools out there, but don't require them to be installed just to run the
> application or use the library; and don't see why users would be
> interested in installing all those just to submit a bug report.

Again:

- the test support tools are only needed to run the test suite, not to
use the library.  I never said otherwise.

- we don't *require* that anyone runs the tests to submit a bug
report, it's just that people often find bugs thanks to the test
suite.  Any report is welcome, of course!

- keep in mind that these tools are typically used for numerical
research, so users do want to see that *on their own computer*, they
can validate certain measure of correctness (not a guarantee, of
course, but seeing a few thousand numerical tests execute correctly
does provide a minimum of confidence).

Ultimately all I can say is that I have *extensive, real-world*
evidence of this approach having worked *fantastically* well for
ipython, numpy, scipy, nipy.*, etc over the last few years.  Others
are free to work as they please, of course.

Regards,

f

ps - I do hope that the new work in unittest2 may make even the nose
dependency perhaps unnecessary for the basic test suite (only to be
used by developers in more detailed work).