[TIP] Test discovery for unittest

[Olemis Lang]

On Mon, Apr 6, 2009 at 4:33 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> On Mon, 2009-04-06 at 16:15 +0100, Michael Foord wrote:
>
>> >  - errors are reported poorly and typically result in the entire
>> test
>> >    suite not running.
>> >
>>
>> How does load_tests alleviate this problem?
>
> I
> would do it differently the second time around, but the code would still
> go in the loader. (Also see Olemis' point about test discovery arguably
> being a loader responsibility - I agree with him about that.

:)

> I don't
> agree about DiscoveringTestSuite, it is just an adapter from loader to a
> memoised test suite, and that's useful)
>

Well, let's see. Can anybody explain me why DiscoveringTestSuite is
useful? I dont understand why a test suite has to be responsible for
loading test cases, especially after reading these [1]_, [2]_.

"""A complete test run has two phases: The first phase is configura-
tion, second phase is test case execution. This may be followed by
analysis and visual display of the results to a user. During
configuration, an object hierarchy of test cases is built,
where one object represents a test case. Any such object is an in-
stance of a subclass of TestCase. Such TestCase subclass instances
(to be validated) that except for the basic domain-specific service
are always leaf objects of the hierarchy. The intermediate nodes
provision through client/service collaborations, all collaborations
and the root node are instances of TestSuite, which mostly provides
the grouping functionality necessary for the tree structure. Both
TestCase and TestSuite implement the Test interface to allow
homogenous treatment of each node in the tree.
"""

dutest is consistent with this: DocTestSuite, the only suite in there
handles the composition of DocTest instances, and the behavior needed
to test'em in an uniform way.

I want u to think, and pls tell me if DiscoveringTestSuite is mixin
phase 1 (carried on by test loaders) and phase 2 (carried on by suites
and test cases), or not. I want u to tell me if this class combines
two orthogonal roles, mixing things and negating traditional XUnit
separation of concerns.

Besides if you take a look at XUnit patterns site [3]_, [4]_ you'll
find out that test discovery is *ALWAYS* performed outside the test
suite (pls take a look at this seriously, there 'r interesting
implementation notes ;) since (I insist) they are orthogonal
functionalities. Oh my ! Cant you just see that?

For example, please tell me how DiscoveringTestSuite will handle the
situations I told you already and the following case :

Consider the following structure :

- mdl.src
  - *.py ( they include my code and doctests  )
- mdl.tests
  - test_*.py ( they include subcls of TestCase and doctests  )
  - web_*.py ( They include web tests )
  - web_*.xml ( They include web tests like those generated by
windmill AFAIK, let's say that they 'r resources declared for each
tests web_* module -the loader takes care of that for you - ;)
  - gui_*.py ( They include GUI tests )
  - fit_*.* ( They include tests written with FITness ...)
  - If you want put more in here ...

In this case using dutest you do:

{{{
#! python

loaders = {
   'mdl.src.*' : unittest.defaultTestLoader,  # Load doctests
   'mdl.tests.test_*' : dutest.defaultTestLoader, # Load both doctests
& unittests
                                  # equivalent to dutest.MultiTestLoader(
                                  #
unittest.defaultTestLoader,
                                  #
DocTestLoader()))
   'mdl.tests.web_*' : dutest.MultiTestLoader(WebTestLoader(),

WindmillLoader(prefix='web_', ext='xml')
   'mdl.tests.gui_*' : GuiTestLoader(),
   'mdl.tests' : FitnessLoader(),
   }

l = dutest.MultiTestLoader(PackageTestLoader(mdl_re, ldr) \
                                             for mdl_re, ldr in
loaders.iteritems())

suite = l.loadTestsFromModule(mdl)

}}}

>>
>> Why can you not customise the suite returned? test_suite is
>> responsible
>> for creating the suite and has complete control over which loader it
>> uses.
>
> You can't customise it externally. There are several reasons to hook
> test loading:
>  - fix up what discovery might do
>  - customise tests [add parameters, multiply by interfaces]

An example of this is DocTestLoader class which accepts further args
in order to control the doctest options and other params to use at
testing-time    ;)

>  - add or remove tests [e.g. bring in tests from a module not in this
>   namespace]
>  - decorate the tests [add skip decorators for instance]
>  - control the test suite to be returned [add a order randomiser for
>   instance]
>
> Manually choosing the test suite is only one reason - and hooks

oh ! hooks once again. Perhaps I'm missing somethin bu XUnit is based
on OO and OO-patterns. So I wonder what is the place of hooks in OO ?
Why not decorators (- the GoF pattern -) ?

>> We should allow test_suite to return None by the way - meaning no
>> tests for the module / package.
>
> Uhm, sure.

Why not just [] - empty list - ?

[...]

I'm skipping a long complicated list of arguments here, whcih
indicates the complexity inherent to this impl, not to mention the
intrincated semantics

Finally. I repeat. Why not to use dutest? What's wrong with it?

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Se retira el BDFL ... ¿El fin de Python 3k?