[TIP] Test discovery for unittest

Olemis Lang olemis at gmail.com
Mon Apr 13 10:22:30 PDT 2009

On Mon, Apr 13, 2009 at 9:02 AM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> Olemis Lang wrote:
>> [snip...]
>>> Anyway - modulo a few details they look similar. This is reassuring as it
>>> means we're probably on the right track.
>> About this I wanted to say a couple of things:
>>  - ... however dutest also includes classes for test discovery, and
>>    the fact is that what you'r doing now is somehow similar to what
>>    has been done there. That's why I told u about considering it for
>>    this purpose.
>>  - The following requirements were considered in order to build that
>>    solution. It'd be nice to mention them, and perhaps debate so as
>>    to get some kind of cosensus, agreement, relevance, and | or
>>    priorities:
>>      - My conclusions : loadTestsFromModule and all the methods
>>        in unittest.TestCase *SHOULD* be enough ;o)
> Enough for what - what is the relevance of this point, how does it differ
> from what I have suggested?

Oh ! sorry. The fact is that I didnt mention that I was thinking once
about including loadTestsFrom(File|Stream) in unittest (e.g. to load
doctest scripts, and so on ... there 'r many examples ;) What I'm
saying here is that it's quite unlikely that further methods be

I apologize. I was sleepy :-/

>>    * The code for test discovery *HAS TO* be external to the
>>      representation of test cases. Retrieving test artifacts is
>>      completely different from specifying what needs to be tested
>>      (even if they have some relationships). Besides, both features
>>      can vary independently.
> Test discovery is external to test cases, agreed.


>>      - My conclusions : Test discovery is a responsibility of test
>>        loaders. Let's build lots of them !
> How does this conclusion follow? Whether it is in a loader, a suite, or some
> other component doesn't seem to matter from your previous statement.

IMHO ... test loaders should handle test discovery, and suites test
groupings + the details involved in providing a uniform interface for
the run method

E.g. in dutest -I hope you dont mind if I insist- DocTestSuite handles
the grouping of instances of DocTestCase -containing each an instance
of doctest.Example- and also handle the new manner to carry out the
test run by checking interactive examples one after the other by
wrapping doctest.DocTestRunner. All this happens disregarding how the
tests are actually discovered and loaded.

Loaders are somewhere else ...

>>    * There two orthogonal dimensions in test discovery :
>>      - Test discovery across a package hierarchy. In this case the
>>        goal is to find out all its subpackages and retrieve the
>>        tests out of there.
>>        * My conclusions : That's what PackageTestLoader is there for ;^)
> And also DiscoveringTestSuite (which may become a loadTestsFromPackage
> method on a loader - I'm still agnostic on this as I don't think it matters
> massively).

As I already mentionned I see no point in including
loadTestsFromPackage  or something similar (if this is a feature under
consideration, perhaps I'm confusing something about this and
imagining things and you were using a metaphore or whatever). There is
no difference between a package and a module in this respect. The only
thing that varies is the strategy employed for retrieving test cases.

The concrete example is the classes in dutest. If you use a regular
loader then tests are retrieved only from a single module, if you use
PackageTestLoader.loadTestsFromModule you retrieve tests from sub-pkgs
and mdls ... aussi

IMHO -1 for loadTestsFromPackage

>>      - Retrieve multiple kinds of tests (e.g. doctests + unittest's)
>>        from a single module.
>>        * My conclusions : That's what MultiTestLoader is there for :^)
> Not at all interested currently.

Possibly ... but in the future this *MAY* be needed since it *MIGHT* be useful

>>    * Keep the design as simple and flexible as possible, thereby
>>      promoting code reuse. Each class *SHOULD* solve a very specific
>>      problem (the more abstract the better), and do it well.
>>      - My conclusions : That's why PackageTestLoader,
>>        MultiTestLoader, and DocTestLoader are separate classes
> DiscoveringTestSuite would be fine on these grounds.

Yes. The only suggestions in this respect I have so far would be to
include find_files and module_names_from_paths in DiscoveringTestSuite
so that subclasses be able to override that behavior, if they want to.

>>    * Since there are many orthogonal behaviors in testing,
>>      Test code *HAS TO* be able to be adapted
>>      to the many different and complex testing scenarios.
>>      - My conclusion 1: Building monolithic classes or relying on
>>        hard-to-extend mechanisms (e.g. hooks -remember ihooks, and
>>        so on-) is not a good choice. OO design and patterns *SHOULD*
>>        be the way (e.g. in fact ihooks and similar solutions were
>>        discarded in favour of PEP 302 -which is in fact an instance
>>        of Chain of Responsibility design pattern, even if most or
>>        part of us dont like the purity associated to «academic»
>>        argumentss-).
> The suggested design allows reuse - and the hooks are specifically to permit
> customization. What *specific* problems do you see the hooks causing?

First of all I'd like to see examples of the detailed use of those
hooks (in fact I honestly think that I've provided more concrete &
specific example usage than what I've seen about hooks and DiscoTS)

> What are you referring to as monolithic? The twenty or so lines of code in
> DiscoveringTestSuite? Moving the same code into a loader wouldn't change
> that.

First of all, my intention was not to rest some value to
DiscoveryTestSuite. My intention was to agree on the reqs needed, and
provide examples of what I was saying by putting dutest as an example.
I didnt mean in the previous few words that DiscoveryTestSuite is
monolithic. I was talking in general terms (no matter what the
solution is ;). I think it would be very nice that, instead of taking
a defensive position, to take a constructive and analytic approach and
put the features supported by *any imaginable candidate* -and dutest
may be one of them ;)- side-by-side and compare'em.

I mean all my conclusions are not related to DiscoveryTestSuite
exactly, but to any possible candidate, and therefore I'd like to c
similar examples, just to compare both of 'em & maybe others.

>>      - My conclusion 2: A practical solution to this requirements is
>>        the use of GoF's Decorator pattern, since it provides such
>>        flexibility and avoids class explotion (e.g. using mixins), and
>>        besides allows to vary the behaviors «at run-time».
>>        That's why you recently needed to supply the loader in to
>>        «DiscoveryTestSuite», and that's why PackageTestLoader,
>>        and MultiTestLoader are both instances of the Decorator
>>        pattern.
> Uhm... what are you talking about?

Well I'll let you know what I'm talking about in detail. I really like
when some somebody tells me «what are you talking about?»

- You have included a loaderClass attr in DiscoveryTestSuite
- There's a curious situation in DiscoveryTestSuite. Usually loaders
know about test suites, but in this case the suite is the one that
keeps a reference to the loader. What I'm saying here is that
DiscoveryTestSuite makes reference to TestLoader class, whereas the
usual way (yes you wanna do it your own way, and that's up to you ;)
is to implement specialized loaders which override suiteClass field.
- How can I use loaders accepting parameters at creation (run) time
(e.g. dutest.DocTestLoader) together with DiscoveryTestSuite? I mean
if you assume that there is a need for loaders in there, I think any
loader *SHOULD* be supported, n'est-ce pas ?

> Honestly I get tired of people saying
> that we should just use their code
> without giving specific examples of how
> it solves problems and what the implementation looks like.

The only thing I want is to help since what I see is that
DiscoveryTestSuite is more and more similiar to PackageTestLoader and
that, eventually, they could be so close that they could be pretty
much the same thing. If you think that's wrong ... there's no problem
for me to be quiet, and «enjoy the silence» ...

Perhaps there's some kind of missunderstanding here. I was just
talking about general reqs, and specific examples considering dutest
to illustrate'em in order to get a consensus about test disco reqs.
Formerly I said that whether you use dutest or not is always up to you

But up to this time I dont know what to say ... if I talk in abstract
terms, you tell me «we dont like academics», if I employ specific
examples you tell me «I get tired of people saying that we should just
use their code without giving specific examples of how it solves
problems and what the implementation looks like.» even if I have
provided some links and concrete code & use cases, and u havent even
seen'em. But anyway ...

> Despite all these words (soooo many words...) I fail to see the differences
> between the two patterns.

ok. so far it seems I have no more time to follow this, so ...

À bientôt !



Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

More information about the testing-in-python mailing list