[TIP] suitability of unittest.py for large-scale tests

[Michael Foord]

holger krekel wrote:
> On Sat, Apr 04, 2009 at 14:23 +0100, Michael Foord wrote:
>   
>> holger krekel wrote:
>>     
>>> [snip...]
>>> I think others did good postings and i agree (and also mailed
>>> so) that the current addCleanup suggestions make some sense to me.
>>> However, in Python voting style i'd say "-0" on it. 
>>>
>>> Why? I tried to give background for my views that 
>>>
>>> 1) unittest.py is maybe not the right place to drive larger scale 
>>> tests. 
>>>       
>> Do you mean that you don't think unittest is suitable for testing of  
>> large applications or large systems?
>>
>> If so then could you elaborate? Of all the Python  test frameworks it is  
>> probably the most proven in this area of course.
>>     
>
> do you mean unittest.py is the most proven in the area
> of large scale tests?  
>
>   

I would think so, although I would have a hard time proving it. By 
virtue of its age and inclusion in the standard library it is almost 
certainly the most widely used

I also have some experience of using it with large test systems. At 
Resolver One we have >130 000 lines of test code built on unittest that 
do unit testing, functional testing and web testing.

> I regard many test frameworks and methods fit for 
> purposes that unittest.py does not really cover. 
> texttest, selenium or doctests are often better suited for
> acceptance or certain functional testing.  

Personally I find doctest fantastic for testing documentation examples 
but an awful general purpose testing tool. I realise that others disagree.

Tools like selenium, windmill and twill are completely orthogonal to 
unittest - unittest does not have *any* web testing capabilities but 
these tools can be driven from unittest to do web testing. We drive 
Selenium from our unittest framework for example.
> and buildbot or
>   
buildbot is also orthogonal - it is a test running framework that works 
fine with unittest.

> py.test better for integration testing.  

:-)

I'm *very* happy that py.test and nose are pushing forward the state of 
the art in Python testing. I don't like some aspects of the patterns 
they make possible myself (moving away from object oriented test 
organisation), but diversity is a good thing. I don't see unittest as 
fundamentally flawed.

What is weak about unittest is the interaction of the various classes - 
results, loaders, runners and the like which make it hard to provide 
generic extensions of unittest. We have managed to extend unittest with 
a custom runner that pushes results to a database - but combining that 
with someone else's custom runner would be hard.

The py.test and nose plugins are something I'd like to look at for 
unittest in the future.

> And the likes of
> Windmill, twill or figleaf help with certain kinds of testing
> that do not depend on JUnit style setups. 
>   

They don't depend on it but nor are they replacements for it.
> I think using Python functions for expressing tests is 
> great but that trying to fit setup/teardown of multiple 
> test resources, databases, processes, network servers etc. 
> into the JUnit setUp/tearDown model too limiting. 
>   

I agree that a class or module level setUp / tearDown for expensive 
fixtures are something that could be introduced into unittest. They do 
violate test isolation though and can encourage the writing of bad 
tests. Definitely something to think about for the future, but not an 
itch I feel the need to scratch at the moment.


> Maybe a bit of personal experience: I have done and seen some
> scarily large and involved setUp/tearDown functions.  With
> inheritance hierarchies and copy-pasted-plus-slight-modified 
> setup code fragments.  With self.__dict__ being more a 
> global-ish name space with multiple places that write to it
> and read from it.  Where for writing a test function I 
> had to keep the setup scenario of my particular test class
> in mind.  Where tests functions became hard to move across 
> classes or files. 
>
> The worst, however, is that when i wanted to change the
> way application or components are initialized or bootstrapped
> and how test resources are setup i had to change a lot of
> places in tests.  Suddenly tests didn't help refactoring, but 
> rather hindered it because even a simple change could lead to 
> many changes in test files and only few changes to app code.  
>
>   

The writing of bad tests is possible in any framework - those sound like 
very bad tests to me. We have several different TestCase base classes 
for different types of tests:

* FunctionalTestCase
* ServerTestCase
* GUIThreadTestCase
* InstallerTestCase

and so on. Each of these have their own setUp and tearDown and some of 
them extend each other - FunctionalTestCase extends GUIThreadTestCase 
for example. We find this object oriented pattern easy to understand and 
extend. As I said, I see weaknesses in unittest elsewhere, but I don't 
think the problems you mention are something that an alternative 
framework would particularly make less likely...

Michael Foord

-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog