[TIP] Test discovery for unittest
Robert Collins
robertc at robertcollins.net
Sat Apr 4 17:38:45 PDT 2009
On Sun, 2009-04-05 at 00:38 +0100, Michael Foord wrote:
> Right - nice. It also lets it be used with a custom test runner.
Yup.
Totally separately, I think we should focus on three protocols -
TestLoader, TestCase and TestResult. TestRunner is more often a nuisance
than anything else IME. (And thats dealing with test suites with many
thousand fixtures).
> > There really should be totally separate introspection facilities for
> > finding python modules and packages. bzr's plugin support would be a lot
> > simpler if we didn't have to look for python files, exclude .pyc in -O
> > mode, exclude .pyo in normal mode etc etc. And the same support logic
> > would be most of the discovery logic.
> >
> Right. I wonder if importlib makes this any easier?
It might, I haven't played with it yet.
> > There is only potential glitch with discovery, and thats packages.
> >
> > Say you discover the tests in foo and foo.bar. If foo defines a test
> > loading hook (like test_suite or load_tests) which explicitly loads the
> > tests from foo.bar, then discovery will find bar's tests twice.
> >
> >
> You mean that if a package explicitly loads tests from a subpackage into
> its namespace?
No. But that is another thing that happens. See below for what I did
mean.
> We could keep track of test classes loaded and drop duplicates.
That might be a good idea for other reasons, but I think its unrelated
to discovery :).
> I've been given the go ahead to apply your patch for __iter__ by the way.
Cool.
> I've modified my recipe to include the DiscoveringTestSuite - need to
> think about a testing strategy.
>
> > I'm not sure of the right answer to that yet :).
> >
> Ignore the problem initially. :-)
Actually I've been prompted to think about it. I think the right answer
is to make the return code of the hook control this. As TestLoader
doesn't support this today anyway its moot for now.
As for what I meant; there are two (that I know of) conventions in a
some of the larger python programs out there. The most common is
'test_suite' or 'testsuite' (spellings vary). e.g. if you run 'trial
foo' and there is a foo.test_suite which is callable, trial will skip
discovery and instead do
suite = foo.test_suite()
So it acts as a hook to control what tests are found.
The load_tests hook, which I added to bzr, is an attempt to reduce
duplication from the test_suite hook, while adding in support for custom
loaders and so on.
Compare:
def test_suite():
loader = unittest.TestLoader()
names = [__name__] + [__name__ + ".tests.test_" + name for name in
[
"bar",
"foo",
"gam",
]]
tests = loader.loadTestsFromNames(names)
# do things to tests here
return tests
with
def load_tests(standard_tests, module, loader):
names = ["tests.test_" + name or name in
[
"bar",
"foo",
"gam",
]]
standard_tests.addTest(loader.loadTestsFromNames(
names, module=module))
# Do things to standard_tests here
return standard_tests
The second one removes all the machinery for choosing a loader (allows
custom loader) and loading the tests in the same module (means that
trivial cases in modules which are just parameterising/customising tests
become really simple).
To fit well with discovery, I think that a hook should be able to
locally control discovery.
So I'd propose two things; a standard way of describing what discovery
should look for without needing the list of include_pattern,
exclude_pattern that may change in future. So perhaps:
class DiscoveryRules:
"""Describes rules for discovery of tests."""
def __init__(self, include_filter, exclude_filter):
self.include_filter = include_filter
self.exclude_filter = exclude_filter
def load_tests(standard_tests, module, loader):
"""Customise the found tests for this module/package.
:param standard_tests: The tests found by loader in this module
:param module: The module object that tests are being loaded from.
:param loader: The test loader being used to find and load tests.
:return: A tuple (discover, test_suite) where:
test_suite are the tests to use for this module.
If module is not a package, the discover element is ignored.
otherwise if discover is a DiscoveryRules instance or True then
test discovery will proceed inside the package. Returning a
DiscoveryRules instance allows control of the discovery within
the package.
"""
The primary point of this is scaling: To allow local description of how
to discover tests without having to specify the loader.
-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090405/8ba28125/attachment-0001.pgp
More information about the testing-in-python
mailing list