[TIP] Test discovery for unittest

[Michael Foord]

Robert Collins wrote:
> On Sun, 2009-04-05 at 00:38 +0100, Michael Foord wrote:
>
>
>   
>> Right - nice. It also lets it be used with a custom test runner.
>>     
>
> Yup.
>
> Totally separately, I think we should focus on three protocols -
> TestLoader, TestCase and TestResult. TestRunner is more often a nuisance
> than anything else IME. (And thats dealing with test suites with many
> thousand fixtures).
>
>   

Hmm... in our framework we have a custom TestResult, but also a 
TestRunner that overrides _makeResult.

Our call to actually run tests looks like:

    PublishingTestRunner(verbosity=2).run(suite)

This pushes results (and tracebacks) into the database as the tests run. 
If we want the TextTestRunner to use our custom result objects then as 
far as I can tell we have to do this?

> [snip...]
>> I've modified my recipe to include the DiscoveringTestSuite - need to 
>> think about a testing strategy.
>>
>>     
>>> I'm not sure of the right answer to that yet :).
>>>   
>>>       
>> Ignore the problem initially. :-)
>>     
>
> Actually I've been prompted to think about it. I think the right answer
> is to make the return code of the hook control this. As TestLoader
> doesn't support this today anyway its moot for now.
> As for what I meant; there are two (that I know of) conventions in a
> some of the larger python programs out there. The most common is
> 'test_suite' or 'testsuite' (spellings vary). e.g. if you run 'trial
> foo' and there is a foo.test_suite which is callable, trial will skip
> discovery and instead do
> suite = foo.test_suite()
>
> So it acts as a hook to control what tests are found.
>
>   

So you're suggesting a protocol for specifying a test suite to use and 
not 'manually' collect tests from a module / package. That seems like 
the right level of customization to provide.

> The load_tests hook, which I added to bzr, is an attempt to reduce
> duplication from the test_suite hook, while adding in support for custom
> loaders and so on.
>
> Compare:
> def test_suite():
>     loader = unittest.TestLoader()
>     names = [__name__] + [__name__ +  ".tests.test_" + name for name in
>         [
>         "bar",
>         "foo",
>         "gam",
>         ]]
>     tests = loader.loadTestsFromNames(names)
>     # do things to tests here
>     return tests
>
>   

So if a module provides 'test_suite' it is a callable that provides the 
tests for that module.

The code above implies that it loads the tests for the whole package - 
so if test_suite is found in a package (the __init__.py) then we stop 
recursing into the package. If test_suite is found in a module then we 
use it for that module but continue searching the package / sub-packages.

(Note: I see below that you suggest controlling whether we recurse into 
packages by having load_tests return two values. I think this is 
starting to get overly complex for an initial version. See below for my 
suggestions.)

> with
> def load_tests(standard_tests, module, loader):
>     names = ["tests.test_" + name or name in 
> 	[
> 	"bar",
>         "foo",
>         "gam",
> 	]]
>     standard_tests.addTest(loader.loadTestsFromNames(
>         names, module=module))
>     # Do things to standard_tests here
>     return standard_tests
>
> The second one removes all the machinery for choosing a loader (allows
> custom loader) 

Why is test_suite not enough? test_suite allows you to use a custom 
loader and anything you do in load_tests to reduce duplication could 
also be done in test_suite.

My initial goal is to provide a simple system that works. Having the 
test_suite protocol seems like it should be enough to allow for 
customisation of module / package loading. Why not have test_suite take 
a loader argument and the implementation of test_suite is free to use it 
or ignore it.


> and loading the tests in the same module (means that
> trivial cases in modules which are just parameterising/customising tests
> become really simple).
>
> To fit well with discovery, I think that a hook should be able to
> locally control discovery.
>
> So I'd propose two things; a standard way of describing what discovery
> should look for without needing the list of include_pattern,
> exclude_pattern that may change in future. So perhaps:
>
> class DiscoveryRules:
>     """Describes rules for discovery of tests."""
>
>     def __init__(self, include_filter, exclude_filter):
>        self.include_filter = include_filter
>        self.exclude_filter = exclude_filter
>
> def load_tests(standard_tests, module, loader):
>     """Customise the found tests for this module/package.
>
>     :param standard_tests: The tests found by loader in this module
>     :param module: The module object that tests are being loaded from.
>     :param loader: The test loader being used to find and load tests.
>     :return: A tuple (discover, test_suite) where:
>         test_suite are the tests to use for this module.
>         If module is not a package, the discover element is ignored.
>         otherwise if discover is a DiscoveryRules instance or True then
>         test discovery will proceed inside the package. Returning a
>         DiscoveryRules instance allows control of the discovery within
>         the package.
>     """
>   

Here you are changing the suggested protocol above so that load_tests 
allows individual *packages* to customize how discovery is done inside 
them (it doesn't seem like that would make sense for a module as you 
could then get multiple different contradictory rulesets within a single 
package).
 
Anyway, at this point we're getting beyond what I want to achieve with 
an initial version. The test_suite protocol seems to allow modules and 
packages to completely customize how tests are loaded from them. Don't 
forget that they can also use their own DiscoveringTestSuite if they 
want to change the filter rules.

So in summary +1 on the test_suite protocol (a callable taking a loader 
as an argument) and -1 on load_tests and the more complex rules.

We can always add load_tests as a next step, whereas if we add it now we 
are tied to forever remaining compatible with the API signature we choose.

Michael

> The primary point of this is scaling: To allow local description of how
> to discover tests without having to specify the loader.
>
> -Rob
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog