[TIP] suitability of unittest.py for large-scale tests

[holger krekel]

On Sat, Apr 04, 2009 at 14:23 +0100, Michael Foord wrote:
> holger krekel wrote:
>> [snip...]
>> I think others did good postings and i agree (and also mailed
>> so) that the current addCleanup suggestions make some sense to me.
>> However, in Python voting style i'd say "-0" on it. 
>>
>> Why? I tried to give background for my views that 
>>
>> 1) unittest.py is maybe not the right place to drive larger scale 
>> tests. 
>
> Do you mean that you don't think unittest is suitable for testing of  
> large applications or large systems?
>
> If so then could you elaborate? Of all the Python  test frameworks it is  
> probably the most proven in this area of course.

do you mean unittest.py is the most proven in the area
of large scale tests?  

I regard many test frameworks and methods fit for 
purposes that unittest.py does not really cover. 
texttest, selenium or doctests are often better suited for
acceptance or certain functional testing.  and buildbot or
py.test better for integration testing.  And the likes of
Windmill, twill or figleaf help with certain kinds of testing
that do not depend on JUnit style setups. 

I think using Python functions for expressing tests is 
great but that trying to fit setup/teardown of multiple 
test resources, databases, processes, network servers etc. 
into the JUnit setUp/tearDown model too limiting. 

Maybe a bit of personal experience: I have done and seen some
scarily large and involved setUp/tearDown functions.  With
inheritance hierarchies and copy-pasted-plus-slight-modified 
setup code fragments.  With self.__dict__ being more a 
global-ish name space with multiple places that write to it
and read from it.  Where for writing a test function I 
had to keep the setup scenario of my particular test class
in mind.  Where tests functions became hard to move across 
classes or files. 

The worst, however, is that when i wanted to change the
way application or components are initialized or bootstrapped
and how test resources are setup i had to change a lot of
places in tests.  Suddenly tests didn't help refactoring, but 
rather hindered it because even a simple change could lead to 
many changes in test files and only few changes to app code.  

> If you don't mean this then I'm afraid I don't understand.
>> 2) improving testing in python is best done outside the standard lib 
>
> And a lot of progress has been made outside of the standard library.  
> Test discovery is badly needed for unittest, and the addCleanup proposal  
> is a relatively small change (and a good one I think) - hardly *major*  
> innovation. :-)
>
>> The "-0" also signals that none of these remarks is meant to
>> prevents changes to unittest.py.  I understand and fully respect
>> that people want to take a pragmatic approach and basically
>> say: so much test code is using unittest.py style tests so
>> let's evolve this standard mechanism so that newcomers have a
>> richer and easier model for writing their tests.  And let's
>> use a method that several test runners (including py.test, on
>> a sidenote :) are successfully using.  Hum, maybe i am
>> actually +0 :)  
>>
>> hope this clarifies things a bit.   
>
> Ha. Cool. I'll take the +0. :-)

heh :) 
cheers,
holger

> Thanks
>
> Michael
>
>> cheers,
>> holger
>>
>>   
>
>
> -- 
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog
>
>

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu