[TIP] Test discovery for unittest
Olemis Lang
olemis at gmail.com
Mon Apr 13 06:44:02 PDT 2009
On Fri, Apr 10, 2009 at 12:13 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> Olemis Lang wrote:
>> On Fri, Apr 10, 2009 at 11:28 AM, Michael Foord
>> <fuzzyman at voidspace.org.uk> wrote:
>>
>>> Olemis Lang wrote:
>>>
>>>>
>>>> [snip...]
>>>>
>>>>>
>>>>> I am considering your point that this functionality better belongs
>>>>> in a loader than a suite though.
>>>>>
>>>>
>>>> In that point ... how different would it be from PackageTestLoader ?
>>>>
>>>
>>> unittest has no PackageTestLoader. I assume you mean one in dutest?
>>> Perhaps
>>> you can tell us how it is different.
>>>
>>>
>>
>> I already provided URL so that u all could see it. Therefore I include
>> it here. Just for the record :
>
> Sorry I didn't see the URL. Anyway - a synposis of the differences rather
> than the code was what I asked for. Jeez, do I have to do all the work. ;-)
>
The fact is that since PackageTestLoader was written some months ago
(and therefore is not derived from DiscoveryTestSuite at all ), I
thought that sending a diff would duplicate the lines since LOCs in
both files would be shown ... ;o)
> Anyway - modulo a few details they look similar. This is reassuring as it
> means we're probably on the right track.
>
About this I wanted to say a couple of things:
- I appreciate your comment about doctest + unittest integration
after unittest test discovery ...
- ... however dutest also includes classes for test discovery, and
the fact is that what you'r doing now is somehow similar to what
has been done there. That's why I told u about considering it for
this purpose.
- The following requirements were considered in order to build that
solution. It'd be nice to mention them, and perhaps debate so as
to get some kind of cosensus, agreement, relevance, and | or
priorities:
* Tests *HAVE TO* be always associated to a module. People writes
tests to ensure that some code provides an expected output
given some inputs. If a test (is | can) not be written directly
inside the module's body, then the tester *HAS TO* include the
appropriate files in the form of resources (provided a suitable
infrastructure for accessing such data in stdlib ... and AFAIK
it's on the way ;).
- My conclusions : loadTestsFromModule and all the methods
in unittest.TestCase *SHOULD* be enough ;o)
* The code for test discovery *HAS TO* be external to the
representation of test cases. Retrieving test artifacts is
completely different from specifying what needs to be tested
(even if they have some relationships). Besides, both features
can vary independently.
- My conclusions : Test discovery is a responsibility of test
loaders. Let's build lots of them !
* There two orthogonal dimensions in test discovery :
- Test discovery across a package hierarchy. In this case the
goal is to find out all its subpackages and retrieve the
tests out of there.
* My conclusions : That's what PackageTestLoader is there for ;^)
- Retrieve multiple kinds of tests (e.g. doctests + unittest's)
from a single module.
* My conclusions : That's what MultiTestLoader is there for :^)
* Keep the design as simple and flexible as possible, thereby
promoting code reuse. Each class *SHOULD* solve a very specific
problem (the more abstract the better), and do it well.
- My conclusions : That's why PackageTestLoader,
MultiTestLoader, and DocTestLoader are separate classes
* Since there are many orthogonal behaviors in testing, the real
value of the former simple classes comes from arranging them
in different manners. Test code *HAS TO* be able to be adapted
to the many different and complex testing scenarios.
- My conclusion 1: Building monolithic classes or relying on
hard-to-extend mechanisms (e.g. hooks -remember ihooks, and
so on-) is not a good choice. OO design and patterns *SHOULD*
be the way (e.g. in fact ihooks and similar solutions were
discarded in favour of PEP 302 -which is in fact an instance
of Chain of Responsibility design pattern, even if most or
part of us dont like the purity associated to «academic»
argumentss-).
- My conclusion 2: A practical solution to this requirements is
the use of GoF's Decorator pattern, since it provides such
flexibility and avoids class explotion (e.g. using mixins), and
besides allows to vary the behaviors «at run-time».
That's why you recently needed to supply the loader in to
«DiscoveryTestSuite», and that's why PackageTestLoader,
and MultiTestLoader are both instances of the Decorator
pattern.
* There should be a separation between what can be tested and
what is gonna be tested at a given moment. It *SHOULD* be
possible to specify «test targets» or «test sets», and be able to load
a subset of all the test artifacts included in such targets | sets.
- For example, there are many complex SUTs (e.g. stdlib). If I
want to perform some regression testing in order to change
something in there, then I'd like to test only the code that
could be affected due to the propagation of a failure caused
by the recent introduction of a defect. A more extensive test
run could be deferred to a later time.
- Another example, during a project lifecycle, it is possible
that a team might want to perform some kind of tests at an
specific moment (e.g. unittest's) and others under other
circumstances (e.g. Fit(Ness), web tests, ...)
- I heard something Michael mentionned before about the
similarity between DiscoveryTestSuite and PackageTestLoader.
In this point there is a (possibly) subtle difference between
them.
* With DiscoveryTestSuite you immediately «load» (lazily ;) the tests,
possibly using recursion across the package's children. So
the goal of using that class is to load the tests right away.
E.g. exceuting the following code yields as a result a test
suite, thereby «loading» test cases:
{{{
#!python
suite = DiscoveringTestSuite(start_dir, include_filter,
exclude_filter, is_top_level)
}}}
* Using PackageTestLoader the goal is to specify *IN DETAIL*
the rules that *HAVE TO* be followed to load a group of
test cases (thereby defining something I call «test
targets | set» which is not exactly the same as SUT, what I call
a «test target | set» is a subset of the whole SUT that needs
to be tested). E.g. executing the following code does not
load any test case at all ...
{{{
#!python
l = dutest.PackageTestLoader('oop\.patterns\.(.*)', dutest.DocTestLoader())
}}}
... in fact, after executing this the tester is able to
load either all doctests inside oop.patterns package ...
{{{
#! python
s = l.loadTestsFromModule(oop.patterns)
}}}
... or only those defined in the GoF catalog or in pattern
catalog X ...
{{{
#!python
gof_suite = l.loadTestsFromModule(oop.patterns._GoF)
x_suite = l.loadTestsFromModule(oop.patterns._CatalogX)
}}}
... and all this using the same loader instance.
* Smooth integration with distutils is a very cool feature.
- Examples of such successful integration to achieve an enhanced
productivity is Babel [1]_ in the field of i18n.
- Since it's possible (and simple) to serialize the hierarchy
of loader instances then then appproach taken by dutest makes
also possible to specify different test targets in a
hierarchical config file for later integration with distutils,
and potentially command line like the following can be
obtained:
{{{
$ setup.py test --target integration --module
mybpms.bpel.ws.(sec|transact|disco|ode)
$ setup.py test --target web_and_unit --module mybpms.bpel.ws.(sec|transact|ode)
$ setup.py test --target integration --module mybpms.admin.plugins.*
$ setup.py test --target accept --module myerp.bussrules.accounting.buss_*
}}}
... even if heterogeneous suites (i.e. different types of
tests) are needed.
And that's enough ... it's 2:30 am (when I wrote this) ... %$ ... hip €× € € ×
PS: I have found some bugs in dutest discovery classes due to some
wrong assumptions I had made when I included those features in the
lib (~= nov 2008). So, now I'm in the process of fixing them. Once I
finish I'll let you know ;o)
I apologize if this message was «a little bit» long, but I think it
may be good, and useful for the current debate in order to come up
with a good and useful solution; and also with a clear idea of the
features that are really needed.
.. [1] Babel
(http://babel.edgewall.org)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
More information about the testing-in-python
mailing list