[TIP] Test discovery for unittest

Olemis Lang olemis at gmail.com
Mon Apr 13 06:44:02 PDT 2009

On Fri, Apr 10, 2009 at 12:13 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> Olemis Lang wrote:
>> On Fri, Apr 10, 2009 at 11:28 AM, Michael Foord
>> <fuzzyman at voidspace.org.uk> wrote:
>>> Olemis Lang wrote:
>>>> [snip...]
>>>>> I am considering your point that this functionality better belongs
>>>>> in a loader than a suite though.
>>>> In that point ... how different would it be from PackageTestLoader ?
>>> unittest has no PackageTestLoader. I assume you mean one in dutest?
>>> Perhaps
>>> you can tell us how it is different.
>> I already provided URL so that u all could see it. Therefore I include
>> it here. Just for the record :
> Sorry I didn't see the URL. Anyway  - a synposis of the differences rather
> than the code was what I asked for. Jeez, do I have to do all the work. ;-)

The fact is that since PackageTestLoader was written some months ago
(and therefore is not derived from DiscoveryTestSuite at all ), I
thought that sending a diff would duplicate the lines since LOCs in
both files would be shown ... ;o)

> Anyway - modulo a few details they look similar. This is reassuring as it
> means we're probably on the right track.

About this I wanted to say a couple of things:

  - I appreciate your comment about doctest + unittest integration
    after unittest test discovery ...
  - ... however dutest also includes classes for test discovery, and
    the fact is that what you'r doing now is somehow similar to what
    has been done there. That's why I told u about considering it for
    this purpose.
  - The following requirements were considered in order to build that
    solution. It'd be nice to mention them, and perhaps debate so as
    to get some kind of cosensus, agreement, relevance, and | or
    * Tests *HAVE TO* be always associated to a module. People writes
      tests to ensure that some code provides an expected output
      given some inputs. If a test (is | can) not be written directly
      inside the module's body, then the tester *HAS TO* include the
      appropriate files in the form of resources (provided a suitable
      infrastructure for accessing such data in stdlib ... and AFAIK
      it's on the way ;).
      - My conclusions : loadTestsFromModule and all the methods
        in unittest.TestCase *SHOULD* be enough ;o)
    * The code for test discovery *HAS TO* be external to the
      representation of test cases. Retrieving test artifacts is
      completely different from specifying what needs to be tested
      (even if they have some relationships). Besides, both features
      can vary independently.
      - My conclusions : Test discovery is a responsibility of test
        loaders. Let's build lots of them !
    * There two orthogonal dimensions in test discovery :
      - Test discovery across a package hierarchy. In this case the
        goal is to find out all its subpackages and retrieve the
        tests out of there.
        * My conclusions : That's what PackageTestLoader is there for ;^)
      - Retrieve multiple kinds of tests (e.g. doctests + unittest's)
        from a single module.
        * My conclusions : That's what MultiTestLoader is there for :^)
    * Keep the design as simple and flexible as possible, thereby
      promoting code reuse. Each class *SHOULD* solve a very specific
      problem (the more abstract the better), and do it well.
      - My conclusions : That's why PackageTestLoader,
        MultiTestLoader, and DocTestLoader are separate classes
    * Since there are many orthogonal behaviors in testing, the real
      value of the former simple classes comes from arranging them
      in different manners. Test code *HAS TO* be able to be adapted
      to the many different and complex testing scenarios.
      - My conclusion 1: Building monolithic classes or relying on
        hard-to-extend mechanisms (e.g. hooks -remember ihooks, and
        so on-) is not a good choice. OO design and patterns *SHOULD*
        be the way (e.g. in fact ihooks and similar solutions were
        discarded in favour of PEP 302 -which is in fact an instance
        of Chain of Responsibility design pattern, even if most or
        part of us dont like the purity associated to «academic»
      - My conclusion 2: A practical solution to this requirements is
        the use of GoF's Decorator pattern, since it provides such
        flexibility and avoids class explotion (e.g. using mixins), and
        besides allows to vary the behaviors «at run-time».
        That's why you recently needed to supply the loader in to
        «DiscoveryTestSuite», and that's why PackageTestLoader,
        and MultiTestLoader are both instances of the Decorator
    * There should be a separation between what can be tested and
      what is gonna be tested at a given moment. It *SHOULD* be
      possible to specify «test targets» or «test sets», and be able to load
      a subset of all the test artifacts included in such targets | sets.
      - For example, there are many complex SUTs (e.g. stdlib). If I
        want to perform some regression testing in order to change
        something in there, then I'd like to test only the code that
        could be affected due to the propagation of a failure caused
        by the recent introduction of a defect. A more extensive test
        run could be deferred to a later time.
      - Another example, during a project lifecycle, it is possible
        that a team might want to perform some kind of tests at an
        specific moment (e.g. unittest's) and others under other
        circumstances (e.g. Fit(Ness), web tests, ...)
      - I heard something Michael mentionned before about the
        similarity between DiscoveryTestSuite and PackageTestLoader.
        In this point there is a (possibly) subtle difference between
        * With DiscoveryTestSuite you immediately «load» (lazily ;) the tests,
          possibly using recursion across the package's children. So
          the goal of using that class is to load the tests right away.
          E.g. exceuting the following code yields as a result a test
          suite, thereby «loading» test cases:

suite = DiscoveringTestSuite(start_dir, include_filter,
exclude_filter, is_top_level)

        * Using PackageTestLoader the goal is to specify *IN DETAIL*
          the rules that *HAVE TO* be followed to load a group of
          test cases (thereby defining something I call «test
          targets | set» which is not exactly the same as SUT, what I call
          a «test target | set» is a subset of the whole SUT that needs
          to be tested). E.g. executing the following code does not
          load any test case at all ...


l = dutest.PackageTestLoader('oop\.patterns\.(.*)', dutest.DocTestLoader())
          ... in fact, after executing this the tester is able to
          load either all doctests inside oop.patterns package ...

#! python
s = l.loadTestsFromModule(oop.patterns)

          ... or only those defined in the GoF catalog or in pattern
          catalog X ...

gof_suite = l.loadTestsFromModule(oop.patterns._GoF)
x_suite = l.loadTestsFromModule(oop.patterns._CatalogX)

          ... and all this using the same loader instance.
    * Smooth integration with distutils is a very cool feature.
      - Examples of such successful integration to achieve an enhanced
        productivity is Babel [1]_ in the field of i18n.
      - Since it's possible (and simple) to serialize the hierarchy
        of loader instances then then appproach taken by dutest makes
        also possible to specify different test targets in a
        hierarchical config file for later integration with distutils,
        and potentially command line like the following can be

$ setup.py test --target integration --module
$ setup.py test --target web_and_unit --module mybpms.bpel.ws.(sec|transact|ode)
$ setup.py test --target integration --module mybpms.admin.plugins.*
$ setup.py test --target accept --module myerp.bussrules.accounting.buss_*

        ... even if heterogeneous suites (i.e. different types of
        tests) are needed.

And that's enough ... it's 2:30 am (when I wrote this) ... %$ ... hip €× € € ×

PS: I have found some bugs in dutest discovery classes due to some
wrong assumptions I had made when I included those features in the
lib (~= nov 2008). So, now I'm in the process of fixing them. Once I
finish I'll let you know ;o)

I apologize if this message was «a little bit» long, but I think it
may be good, and useful for the current debate in order to come up
with a good and useful solution; and also with a clear idea of the
features that are really needed.

.. [1] Babel



Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

More information about the testing-in-python mailing list