[TIP] unittest q: keeping track of the *right* number of tests

[laidler at stsci.edu]

>
>On Mon, Mar 23, 2009 at 09:26:49AM -0400, laidler at stsci.edu wrote:
>-> >On Sun, Mar 22, 2009 at 4:15 PM, C. Titus Brown <ctb at msu.edu> wrote:
>-> >> Hi all,
>-> >>
>-> >> we're running into an interesting problem over on the pygr project: the
>-> >> number of tests to run varies according to the installation, and we're
>-> >> having trouble tracking the expected number of tests.
>-> >>
>-> >> Briefly, we include or skip entire test suites depending on whether or
>-> >> not MySQL, sqlite, and/or BLAST (a sequence comparison package) are
>-> >> installed.  How should we track how many tests *should* actually be run?
>-> >>
>-> >> (This is a more general problem, incidentally; I've noticed that in
>-> >> projects with thousands of tests, a few can slip through the cracks and
>-> >> become "not executed" without anyone noticing, for a variety of reasons
>-> >> (inappropriate skipping, poor if statement usage, function renaming,
>-> >> etc.)
>-> 
>-> The solution we have adopted is to build this into a separate
>-> test report system, which remembers the names of all the tests
>-> that have previously been run in a given environment and 
>-> reports any missing tests.
>
>Hi Vicki,
>
>this is the slightly more intensive version of what I was thinking
>about.  Do you have any code or (perhaps even more importantly ;) tips &
>tricks to share with us?

Hi Titus,

The code is a bit hackish right now, but at least
some pieces of it will be in a shareable state in the relatively 
near future. 

I will be happy to talk about what we've done at the
testing BoF. :)

Vicki