[TIP] unittest q: keeping track of the *right* number of tests

Marius Gedminas marius at gedmin.as
Mon Mar 23 14:14:25 PDT 2009


On Sun, Mar 22, 2009 at 07:15:08AM -0700, C. Titus Brown wrote:
> we're running into an interesting problem over on the pygr project: the
> number of tests to run varies according to the installation, and we're
> having trouble tracking the expected number of tests.
> 
> Briefly, we include or skip entire test suites depending on whether or
> not MySQL, sqlite, and/or BLAST (a sequence comparison package) are
> installed.  How should we track how many tests *should* actually be run?
> 
> (This is a more general problem, incidentally; I've noticed that in
> projects with thousands of tests, a few can slip through the cracks and
> become "not executed" without anyone noticing, for a variety of reasons
> (inappropriate skipping, poor if statement usage, function renaming,
> etc.)
...
> Thoughts?

I dislike the idea of having to increment a number somewhere when I add
a test.  Then again I was never bitten hard by tests that quietly
slipped through the cracks.

I running pyflakes on the source tree (in buildbot) and watch for
redefined function warnings.  Those have happened a few times, usually
when you copy and paste a test and then forget to change the name.

Having test coverage reports also helps discover tests that somehow got
disabled.

Having discipline to see a test fail on your machine before you make it
pass and check in helps; then you could track (in buildbot) the tests
that were run in the previous run and the ones that are being run now
and warn about ones that disappeared.

I've also experimented with buildbot displaying the number of tests that
have been run: http://zope3.pov.lt/buildbot/ .  It is interesting to
observe the differences in the total number of tests across platforms,
but I don't think anybody ever noticed any tests that got lost by
comparing the total number of tests between builds.

Marius Gedminas
-- 
This is an object-oriented system.  If we change anything, the users object.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090323/d736d6e4/attachment.pgp 


More information about the testing-in-python mailing list