[TIP] Test discovery for unittest

[Marius Gedminas]

On Tue, Apr 07, 2009 at 08:31:42AM +1000, Robert Collins wrote:
> On Mon, 2009-04-06 at 23:17 +0100, Michael Foord wrote:
> > I'm happy with a package / module level hook that takes loader and tests 
> > arguments and is called load_tests or test_suite.
> > 
> > It should return a test suite or None, and if available at the package 
> > level the current implementation won't continue discovery into the 
> > package (allowing a different return value at some future point to 
> > modify discovery for the package would be possible).
> > 
> > Now one of us needs to implement it. :-)
> 
> I will happily do a patch up. Easter is coming up :).
> 
> This has been largely a you-and-I discussion, does anyone else have
> things they want that are not satisfied by this?

I don't think I can give this discussion the full attention it deserves,
but I'll at least outline the conventions that the Zope 3 world uses for
tests:

    * the test runner finds test modules (looking for files named
      'tests.py' or 'tests/test_*.py' in the tree recursively)
    * every test module defines a function test_suite() that takes no
      arguments and returns a unittest.TestSuite() or a subclass
    * some of the test modules return doctest.DocTestSuite(), some of
      the modules do

        def test_suite():
            return unittest.TestSuite([
                        doctest.DocTestSuite(),
                        doctest.DocTestSuite(),
                        doctest.DocFileSuite(),
                        doctest.DocFileSuite(),
                ])

      some older ones also include several test suites constructed with
      unittest.makeSuite(TestCaseClass) into the containing test suite.
    * import (and other test loading) errors are handled by the test runner:
      i.e. if you cannot import test_foo or if test_foo.test_suite()
      raises an exception, you get to see the traceback of it, but that
      does not inhibit the other test modules from getting loaded.

Is this compatible with what you're discussing?  I'm kind of wondering
about the talk about controlling recursion.

There is a desire in the Zope 3 world to migrate away from our own test
runner to something more standard (probably nose), but nobody has the
time to work on that (and we have features---such as test layers with
a single expensive setUp/tearDown function pair wrapping multiple
tests, with the tests grouped into layers according to the 'layer'
attribute of the individual testcases/suites coming from various
locations in the tree---that AFAIK nose doesn't have).

> The following is the guts of the patch: everything else is tests and
> documentation.
> 
>  * Alter TestLoader.loadTestsFromModule:
>         tests = []
>         for name in dir(module):
>             obj = getattr(module, name)
>             if (isinstance(obj, (type, types.ClassType)) and
>                 issubclass(obj, TestCase)):
>                 tests.append(self.loadTestsFromTestCase(obj))
> +        result = self.suiteClass(tests)
> +        load_hook = getattr(module, 'load_tests', None)
> +        if load_hook is not None:
> +            result = load_hook(self, result)
> +        if not result:
> +            return self.suiteClass()
> +        else:
> +            return result
> -        return self.suiteClass(tests)
> 
> 
> (I've chosen load_tests because AFAIK its only used by bzr at the
> moment, and this prevent causing trouble for trial and other general
> purpose test environments that do use test_suite at the moment with a
> different signature.)

Zope 3 doesn't use loadTestsFromModule, so this change won't break
anything.

Marius Gedminas
-- 
Perl is hard for most people to write. They write PERL or Pearl.
        -- Abigail
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090407/7b1c4b73/attachment-0001.pgp