[TIP] Test discovery for unittest

[Michael Foord]

Olemis Lang wrote:
> [snip...]
>> Anyway - modulo a few details they look similar. This is reassuring as it
>> means we're probably on the right track.
>>
>>     
>
> About this I wanted to say a couple of things:
>
>   - I appreciate your comment about doctest + unittest integration
>     after unittest test discovery ...
>   

Good - so it isn't something I want to discuss at the moment, and 
probably won't ever implement myself as I don't use doctest.

>   - ... however dutest also includes classes for test discovery, and
>     the fact is that what you'r doing now is somehow similar to what
>     has been done there. That's why I told u about considering it for
>     this purpose.
>   - The following requirements were considered in order to build that
>     solution. It'd be nice to mention them, and perhaps debate so as
>     to get some kind of cosensus, agreement, relevance, and | or
>     priorities:
>     * Tests *HAVE TO* be always associated to a module. People writes
>       tests to ensure that some code provides an expected output
>       given some inputs. If a test (is | can) not be written directly
>       inside the module's body, then the tester *HAS TO* include the
>       appropriate files in the form of resources (provided a suitable
>       infrastructure for accessing such data in stdlib ... and AFAIK
>       it's on the way ;).
>       - My conclusions : loadTestsFromModule and all the methods
>         in unittest.TestCase *SHOULD* be enough ;o)
>   

Enough for what - what is the relevance of this point, how does it 
differ from what I have suggested?

>     * The code for test discovery *HAS TO* be external to the
>       representation of test cases. Retrieving test artifacts is
>       completely different from specifying what needs to be tested
>       (even if they have some relationships). Besides, both features
>       can vary independently.
>   

Test discovery is external to test cases, agreed.
>       - My conclusions : Test discovery is a responsibility of test
>         loaders. Let's build lots of them !
>   

How does this conclusion follow? Whether it is in a loader, a suite, or 
some other component doesn't seem to matter from your previous statement.

>     * There two orthogonal dimensions in test discovery :
>       - Test discovery across a package hierarchy. In this case the
>         goal is to find out all its subpackages and retrieve the
>         tests out of there.
>         * My conclusions : That's what PackageTestLoader is there for ;^)
>   

And also DiscoveringTestSuite (which may become a loadTestsFromPackage 
method on a loader - I'm still agnostic on this as I don't think it 
matters massively).
>       - Retrieve multiple kinds of tests (e.g. doctests + unittest's)
>         from a single module.
>         * My conclusions : That's what MultiTestLoader is there for :^)
>   

Not at all interested currently.

>     * Keep the design as simple and flexible as possible, thereby
>       promoting code reuse. Each class *SHOULD* solve a very specific
>       problem (the more abstract the better), and do it well.
>       - My conclusions : That's why PackageTestLoader,
>         MultiTestLoader, and DocTestLoader are separate classes
>   

DiscoveringTestSuite would be fine on these grounds.

>     * Since there are many orthogonal behaviors in testing, the real
>       value of the former simple classes comes from arranging them
>       in different manners. Test code *HAS TO* be able to be adapted
>       to the many different and complex testing scenarios.
>       - My conclusion 1: Building monolithic classes or relying on
>         hard-to-extend mechanisms (e.g. hooks -remember ihooks, and
>         so on-) is not a good choice. OO design and patterns *SHOULD*
>         be the way (e.g. in fact ihooks and similar solutions were
>         discarded in favour of PEP 302 -which is in fact an instance
>         of Chain of Responsibility design pattern, even if most or
>         part of us dont like the purity associated to «academic»
>         argumentss-).
>   

The suggested design allows reuse - and the hooks are specifically to 
permit customization. What *specific* problems do you see the hooks causing?

What are you referring to as monolithic? The twenty or so lines of code 
in DiscoveringTestSuite? Moving the same code into a loader wouldn't 
change that.

>       - My conclusion 2: A practical solution to this requirements is
>         the use of GoF's Decorator pattern, since it provides such
>         flexibility and avoids class explotion (e.g. using mixins), and
>         besides allows to vary the behaviors «at run-time».
>         That's why you recently needed to supply the loader in to
>         «DiscoveryTestSuite», and that's why PackageTestLoader,
>         and MultiTestLoader are both instances of the Decorator
>         pattern.
>   

Uhm... what are you talking about?  Honestly I get tired of people 
saying that we should just use their code without giving specific 
examples of how it solves problems and what the implementation looks like.

The examples you give below, as far as I can tell, show no practical 
differences between the two approaches.


>     * There should be a separation between what can be tested and
>       what is gonna be tested at a given moment. It *SHOULD* be
>       possible to specify «test targets» or «test sets», and be able to load
>       a subset of all the test artifacts included in such targets | sets.
>       - For example, there are many complex SUTs (e.g. stdlib). If I
>         want to perform some regression testing in order to change
>         something in there, then I'd like to test only the code that
>         could be affected due to the propagation of a failure caused
>         by the recent introduction of a defect. A more extensive test
>         run could be deferred to a later time.
>       - Another example, during a project lifecycle, it is possible
>         that a team might want to perform some kind of tests at an
>         specific moment (e.g. unittest's) and others under other
>         circumstances (e.g. Fit(Ness), web tests, ...)
>       - I heard something Michael mentionned before about the
>         similarity between DiscoveryTestSuite and PackageTestLoader.
>         In this point there is a (possibly) subtle difference between
>         them.
>         * With DiscoveryTestSuite you immediately «load» (lazily ;) the tests,
>           possibly using recursion across the package's children. So
>           the goal of using that class is to load the tests right away.
>           E.g. exceuting the following code yields as a result a test
>           suite, thereby «loading» test cases:
>
> {{{
> #!python
> suite = DiscoveringTestSuite(start_dir, include_filter,
> exclude_filter, is_top_level)
> }}}
>
>         * Using PackageTestLoader the goal is to specify *IN DETAIL*
>           the rules that *HAVE TO* be followed to load a group of
>           test cases (thereby defining something I call «test
>           targets | set» which is not exactly the same as SUT, what I call
>           a «test target | set» is a subset of the whole SUT that needs
>           to be tested). E.g. executing the following code does not
>           load any test case at all ...
>
> {{{
> #!python
>
> l = dutest.PackageTestLoader('oop\.patterns\.(.*)', dutest.DocTestLoader())
> }}}
>           ... in fact, after executing this the tester is able to
>           load either all doctests inside oop.patterns package ...
>
> {{{
> #! python
> s = l.loadTestsFromModule(oop.patterns)
> }}}
>
>           ... or only those defined in the GoF catalog or in pattern
>           catalog X ...
>
> {{{
> #!python
> gof_suite = l.loadTestsFromModule(oop.patterns._GoF)
> x_suite = l.loadTestsFromModule(oop.patterns._CatalogX)
> }}}
>
>           ... and all this using the same loader instance.
>   


Despite all these words (soooo many words...) I fail to see the 
differences between the two patterns.

Even if we go with discovery in the suite you can still use a specific 
loader for a single module - or start discovery at a specific place in 
the package - which is what your other examples show.

All the best,

Michael

>     * Smooth integration with distutils is a very cool feature.
>       - Examples of such successful integration to achieve an enhanced
>         productivity is Babel [1]_ in the field of i18n.
>       - Since it's possible (and simple) to serialize the hierarchy
>         of loader instances then then appproach taken by dutest makes
>         also possible to specify different test targets in a
>         hierarchical config file for later integration with distutils,
>         and potentially command line like the following can be
>         obtained:
>
> {{{
> $ setup.py test --target integration --module
> mybpms.bpel.ws.(sec|transact|disco|ode)
> $ setup.py test --target web_and_unit --module mybpms.bpel.ws.(sec|transact|ode)
> $ setup.py test --target integration --module mybpms.admin.plugins.*
> $ setup.py test --target accept --module myerp.bussrules.accounting.buss_*
> }}}
>
>         ... even if heterogeneous suites (i.e. different types of
>         tests) are needed.
>
> And that's enough ... it's 2:30 am (when I wrote this) ... %$ ... hip €× € € ×
>
> PS: I have found some bugs in dutest discovery classes due to some
> wrong assumptions I had made when I included those features in the
> lib (~= nov 2008). So, now I'm in the process of fixing them. Once I
> finish I'll let you know ;o)
>
> I apologize if this message was «a little bit» long, but I think it
> may be good, and useful for the current debate in order to come up
> with a good and useful solution; and also with a clear idea of the
> features that are really needed.
>
> .. [1] Babel
>        (http://babel.edgewall.org)
>
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog