[TIP] Test discovery for unittest

[Michael Foord]

Olemis Lang wrote:
> On Fri, Apr 10, 2009 at 8:44 AM, Michael Foord
> <fuzzyman at voidspace.org.uk> wrote:
>   
>> C. Titus Brown wrote:
>>     
>>> On Tue, Apr 07, 2009 at 09:33:56AM -0500, Olemis Lang wrote:
>>> -> I want u to think, and pls tell me if DiscoveringTestSuite is mixin
>>> -> phase 1 (carried on by test loaders) and phase 2 (carried on by suites
>>> -> and test cases), or not. I want u to tell me if this class combines
>>> -> two orthogonal roles, mixing things and negating traditional XUnit
>>> -> separation of concerns.
>>>
>>> Hey Olemis,
>>>
>>> just one quick comment to cut through the thicket of
>>> challenge/responses:
>>>
>>> Practicality beats purity.  Quoting XUnit Patterns to me and/or basing
>>> your discussion on nomenclature taken from XUnit Patterns isn't going to
>>> convince me ;).  Actual examples of screwups caused by such decisions
>>> are more interesting than citations, at least to me.
>>>
>>>       
>> Me too I'm afraid (at least I'm not even trying to be an academic though!).
>> I have *no* interest in keeping unittest to some notional idea of xUnit
>> purity, but in developing it to be a practical tool for testing in Python.
>>
>>     
>
> Oh yes ... good point, and I agree for the second time ;)
>
> In fact I would like to know how useful DiscoveryTestSuite is if the
> package is inside a zip file, or if I distribute it inside a JAR file
> for using it in Jython, or if the same JAR is inside another WAR, EAR,
> or if suddenly I prefer to package Py mdls inside RAR | BZ2 | <add
> your own compressed format here> files ... or if suddenly another
> packaging scheme is adopted, or even if somebody wants to retrieve the
> mdls from other sources (e.g. HTTP server in an intranet -urlimport-),
> or, or, or ...
>
>   

Test discovery is inherently from the filesystem and there is no 
intention for it to work in the scenarios you have described - at least 
initially.

urlimport is a bad idea and we should go out of our way not to support 
it. ;-)

I am considering your point that this functionality better belongs in a 
loader than a suite though.

> IMHO a framework should be built to encourage reusability and be
> abstract enough so that it be useful also if settings, deploy envs and
> so on change a little or too much. I dont see that kind of
> adaptability and fexibility in DiscoveryTestSuite. In the case, of
> dutest, people will always be able to code their own loader and
> perform test discovery from the outside (BTW just like XUnit patterns
> say).
>   

Why is reusability not in the discovering suite as currently implemented?

There is nothing in it that stops people subclassing, or using it in 
conjunction with custom loaders, test runners and test results - which 
are the typical points of customization in unittest.


> It is possible to talk about practicality «in abstract» (so far IMHO
> without the concrete arguments that could make me wonder ;), or have
> some gut feelings against something; anyway I'm sure of the fact that
> nobody'd drive a car made by an amateur in an F1 tournament, and to
> build that car, well you need a lot of engineers, having strong
> academic knowledge about their own field, but also be practical enough
> so that he/she wont need to reinvent Classic Mechanics, and also to
> *MAKE IT WORK* under the many daily circumstances it could face (if I
> need to change the wheels, I dont want to go to the factory -I dont
> know the exact jargon employed in automotive industry, so please, dont
> you ever buy any car that I've built by myself-).
>
>   
>> I think some of the proposals currently advanced (all of which come from
>> practical experience in other Python testing frameworks) go a long way
>> towards that.
>>
>>     
>
> It will always be up to all of you, I think some things in dutest are
> valuable. I also think that it could be helpful for many people and
> IMHO it allows to customize all the aspects provided by doctest
> thereby making possible to use all the features supported by doctest
> backend (which is not possible with doctest unittest API). If you ever
> agree with this and feel that including it in stdlib would be
> beneficial, well, here I am.
>
> IMO also there are many other use cases for doctests and thereby also
> IMO there's a lot of work to do in dutest. Unfortunately I dont have
> enough time :-/
>   


I'm sure there are many good things in dutest. Integration of doctest 
and unittest is something worth looking at in the future but is not 
something I have bandwidth for at the moment. (Particularly as I 
*personally* feel that docttest is a horrible unit testing tool - but 
excellent for testing examples in documentation.)

FWIW this is a subject that Barry Warsaw has expressed interest in. 
*After* we have test discovery in unittest then adding discovery and 
running of doctests for a project is a natural extension. Note the 
*after* though... :-)

Michael

> ... preparing to hibernate ...
>
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog