[TIP] Test discovery for unittest

[Michael Foord]

Marius Gedminas wrote:
> On Tue, Apr 07, 2009 at 08:31:42AM +1000, Robert Collins wrote:
>   
>> On Mon, 2009-04-06 at 23:17 +0100, Michael Foord wrote:
>>     
>>> I'm happy with a package / module level hook that takes loader and tests 
>>> arguments and is called load_tests or test_suite.
>>>
>>> It should return a test suite or None, and if available at the package 
>>> level the current implementation won't continue discovery into the 
>>> package (allowing a different return value at some future point to 
>>> modify discovery for the package would be possible).
>>>
>>> Now one of us needs to implement it. :-)
>>>       
>> I will happily do a patch up. Easter is coming up :).
>>
>> This has been largely a you-and-I discussion, does anyone else have
>> things they want that are not satisfied by this?
>>     
>
> I don't think I can give this discussion the full attention it deserves,
> but I'll at least outline the conventions that the Zope 3 world uses for
> tests:
>
>     * the test runner finds test modules (looking for files named
>       'tests.py' or 'tests/test_*.py' in the tree recursively)
>     * every test module defines a function test_suite() that takes no
>       arguments and returns a unittest.TestSuite() or a subclass
>     * some of the test modules return doctest.DocTestSuite(), some of
>       the modules do
>
>         def test_suite():
>             return unittest.TestSuite([
>                         doctest.DocTestSuite(),
>                         doctest.DocTestSuite(),
>                         doctest.DocFileSuite(),
>                         doctest.DocFileSuite(),
>                 ])
>
>       some older ones also include several test suites constructed with
>       unittest.makeSuite(TestCaseClass) into the containing test suite.
>     * import (and other test loading) errors are handled by the test runner:
>       i.e. if you cannot import test_foo or if test_foo.test_suite()
>       raises an exception, you get to see the traceback of it, but that
>       does not inhibit the other test modules from getting loaded.
>
> Is this compatible with what you're discussing?  I'm kind of wondering
> about the talk about controlling recursion.
>   

It sounds *similar*, except:

- Test modules can be discovered from the file system by a loader / 
custom suite
- Tests are loaded in the normal unittest way - by finding all TestCase 
subclasses in each module found
- Individual modules can optionally define 'load_tests' in which case 
that is called to get the suite from the module
- If a package (in the __init__.py) defines 'load_tests' then that is 
used to load the suite for the whole package and discovery does not 
continue into the package directory (this is the 'controlling recursion' 
business

I disagree that failing to load modules should not be a fatal error by 
default. If a test system wants that behavior it can implement it itself 
in load_tests.

> There is a desire in the Zope 3 world to migrate away from our own test
> runner to something more standard (probably nose), but nobody has the
> time to work on that (and we have features---such as test layers with
> a single expensive setUp/tearDown function pair wrapping multiple
> tests, with the tests grouped into layers according to the 'layer'
> attribute of the individual testcases/suites coming from various
> locations in the tree---that AFAIK nose doesn't have).
>
>   

There is nothing currently proposed for this kind of use case.

Nose does support class and module level setUp and tearDown.

Michael

>> The following is the guts of the patch: everything else is tests and
>> documentation.
>>
>>  * Alter TestLoader.loadTestsFromModule:
>>         tests = []
>>         for name in dir(module):
>>             obj = getattr(module, name)
>>             if (isinstance(obj, (type, types.ClassType)) and
>>                 issubclass(obj, TestCase)):
>>                 tests.append(self.loadTestsFromTestCase(obj))
>> +        result = self.suiteClass(tests)
>> +        load_hook = getattr(module, 'load_tests', None)
>> +        if load_hook is not None:
>> +            result = load_hook(self, result)
>> +        if not result:
>> +            return self.suiteClass()
>> +        else:
>> +            return result
>> -        return self.suiteClass(tests)
>>
>>
>> (I've chosen load_tests because AFAIK its only used by bzr at the
>> moment, and this prevent causing trouble for trial and other general
>> purpose test environments that do use test_suite at the moment with a
>> different signature.)
>>     
>
> Zope 3 doesn't use loadTestsFromModule, so this change won't break
> anything.
>
> Marius Gedminas
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog