[TIP] unittest q: keeping track of the *right* number of tests

Brett Cannon brett at python.org
Sun Mar 22 10:07:55 PDT 2009

On Sun, Mar 22, 2009 at 07:15, C. Titus Brown <ctb at msu.edu> wrote:

> Hi all,
> we're running into an interesting problem over on the pygr project: the
> number of tests to run varies according to the installation, and we're
> having trouble tracking the expected number of tests.
> Briefly, we include or skip entire test suites depending on whether or
> not MySQL, sqlite, and/or BLAST (a sequence comparison package) are
> installed.  How should we track how many tests *should* actually be run?
> (This is a more general problem, incidentally; I've noticed that in
> projects with thousands of tests, a few can slip through the cracks and
> become "not executed" without anyone noticing, for a variety of reasons
> (inappropriate skipping, poor if statement usage, function renaming,
> etc.)
> The two ideas I had are --
>  - keep track of aggregate number of tests per test file expected under
>    the various conditions, e.g.
>        test_db_stuff: 20 tests if no MySQL, 50 if MySQL
>  - keep track of individual tests by name, and tag them with "not run
>    if MySQL is not installed", and then check to make sure all of the
>    expected ones are run.
> Both entail an increase in record-keeping which is annoying but
> inevitable, I guess...
> Thoughts?

First issue that comes to mind is you will obviously have to clearly state
what potential resources or variable could be lacking to cause a test to not
be executed. That can be a chore in itself.

Second, depending on how your tests are written, you could potentially
automate a decent amount of the discovery through decorators. If you have
decorators flag what tests require what settings you can then import a
module and have the decorators simply record the details needed to run that
test. This obviously has a potential issue of errors in mis-stating what the
test really needs, but that could happen with writing them down by hand

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.idyll.org/pipermail/testing-in-python/attachments/20090322/4f602edb/attachment.html 

More information about the testing-in-python mailing list