[TIP] Test discovery for unittest

Robert Collins robertc at robertcollins.net
Sat Apr 4 17:38:45 PDT 2009


On Sun, 2009-04-05 at 00:38 +0100, Michael Foord wrote:


> Right - nice. It also lets it be used with a custom test runner.

Yup.

Totally separately, I think we should focus on three protocols -
TestLoader, TestCase and TestResult. TestRunner is more often a nuisance
than anything else IME. (And thats dealing with test suites with many
thousand fixtures).

> > There really should be totally separate introspection facilities for
> > finding python modules and packages. bzr's plugin support would be a lot
> > simpler if we didn't have to look for python files, exclude .pyc in -O
> > mode, exclude .pyo in normal mode etc etc. And the same support logic
> > would be most of the discovery logic.
> >   
> Right. I wonder if importlib makes this any easier?

It might, I haven't played with it yet.

> > There is only potential glitch with discovery, and thats packages.
> >
> > Say you discover the tests in foo and foo.bar. If foo defines a test
> > loading hook (like test_suite or load_tests) which explicitly loads the
> > tests from foo.bar, then discovery will find bar's tests twice.
> >
> >   
> You mean that if a package explicitly loads tests from a subpackage into 
> its namespace?

No. But that is another thing that happens. See below for what I did
mean.

> We could keep track of test classes loaded and drop duplicates.

That might be a good idea for other reasons, but I think its unrelated
to discovery :).

> I've been given the go ahead to apply your patch for __iter__ by the way.

Cool.

> I've modified my recipe to include the DiscoveringTestSuite - need to 
> think about a testing strategy.
> 
> > I'm not sure of the right answer to that yet :).
> >   
> Ignore the problem initially. :-)

Actually I've been prompted to think about it. I think the right answer
is to make the return code of the hook control this. As TestLoader
doesn't support this today anyway its moot for now.

As for what I meant; there are two (that I know of) conventions in a
some of the larger python programs out there. The most common is
'test_suite' or 'testsuite' (spellings vary). e.g. if you run 'trial
foo' and there is a foo.test_suite which is callable, trial will skip
discovery and instead do
suite = foo.test_suite()

So it acts as a hook to control what tests are found.

The load_tests hook, which I added to bzr, is an attempt to reduce
duplication from the test_suite hook, while adding in support for custom
loaders and so on.

Compare:
def test_suite():
    loader = unittest.TestLoader()
    names = [__name__] + [__name__ +  ".tests.test_" + name for name in
        [
        "bar",
        "foo",
        "gam",
        ]]
    tests = loader.loadTestsFromNames(names)
    # do things to tests here
    return tests

with
def load_tests(standard_tests, module, loader):
    names = ["tests.test_" + name or name in 
	[
	"bar",
        "foo",
        "gam",
	]]
    standard_tests.addTest(loader.loadTestsFromNames(
        names, module=module))
    # Do things to standard_tests here
    return standard_tests

The second one removes all the machinery for choosing a loader (allows
custom loader) and loading the tests in the same module (means that
trivial cases in modules which are just parameterising/customising tests
become really simple).

To fit well with discovery, I think that a hook should be able to
locally control discovery.

So I'd propose two things; a standard way of describing what discovery
should look for without needing the list of include_pattern,
exclude_pattern that may change in future. So perhaps:

class DiscoveryRules:
    """Describes rules for discovery of tests."""

    def __init__(self, include_filter, exclude_filter):
       self.include_filter = include_filter
       self.exclude_filter = exclude_filter


def load_tests(standard_tests, module, loader):
    """Customise the found tests for this module/package.

    :param standard_tests: The tests found by loader in this module
    :param module: The module object that tests are being loaded from.
    :param loader: The test loader being used to find and load tests.
    :return: A tuple (discover, test_suite) where:
        test_suite are the tests to use for this module.
        If module is not a package, the discover element is ignored.
        otherwise if discover is a DiscoveryRules instance or True then
        test discovery will proceed inside the package. Returning a
        DiscoveryRules instance allows control of the discovery within
        the package.
    """

The primary point of this is scaling: To allow local description of how
to discover tests without having to specify the loader.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090405/8ba28125/attachment-0001.pgp 


More information about the testing-in-python mailing list