[TIP] suitability of unittest.py for large-scale tests

Sat Apr 4 10:34:45 PDT 2009

Hi Michael, 

thanks for your mail.  I think we agree that there are a
lot of good test tools, methods and libs out which are mostly 
orthogonal to unittest.py.  I understand that unittest.py can
be extended through subclasses in order to deal with all kinds
of testing.  And that you and others have successfully done
so.  And that you want to work on improving unittest. 

I am glad to see that you are happily hacking on unittest.  
In fact, i may well want to integrate the results of your work
into a py.test release so that people can use the new
mechanisms rather sooner and cross-python version. 

My current goal is to have all kinds of test libs and 
test types available at a fingertipp.  So I am currently hacking
on tools to collect, run and debug all kinds of tests, up to eventually
integrating with actual deployment.  I eventually see deployment 
as another kind of test run where i want to have helpful information 
when things go wrong :)  Much of this certainly is, as you point out, 
orthogonal to what unittest.py tries to provide.  

For me it's fine to leave it at that for now. 

cheers and have fun, 
holger

Sat, Apr 04, 2009 at 17:18 +0100, Michael Foord wrote:
> holger krekel wrote:
>> On Sat, Apr 04, 2009 at 14:23 +0100, Michael Foord wrote:
>>   
>>> holger krekel wrote:
>>>     
>>>> [snip...]
>>>> I think others did good postings and i agree (and also mailed
>>>> so) that the current addCleanup suggestions make some sense to me.
>>>> However, in Python voting style i'd say "-0" on it. 
>>>>
>>>> Why? I tried to give background for my views that 
>>>>
>>>> 1) unittest.py is maybe not the right place to drive larger scale  
>>>> tests.       
>>> Do you mean that you don't think unittest is suitable for testing of  
>>> large applications or large systems?
>>>
>>> If so then could you elaborate? Of all the Python  test frameworks it 
>>> is  probably the most proven in this area of course.
>>>     
>>
>> do you mean unittest.py is the most proven in the area
>> of large scale tests?  
>>
>>   
>
> I would think so, although I would have a hard time proving it. By  
> virtue of its age and inclusion in the standard library it is almost  
> certainly the most widely used
>
> I also have some experience of using it with large test systems. At  
> Resolver One we have >130 000 lines of test code built on unittest that  
> do unit testing, functional testing and web testing.
>
>> I regard many test frameworks and methods fit for purposes that 
>> unittest.py does not really cover. texttest, selenium or doctests are 
>> often better suited for
>> acceptance or certain functional testing.  
>
> Personally I find doctest fantastic for testing documentation examples  
> but an awful general purpose testing tool. I realise that others 
> disagree.
>
> Tools like selenium, windmill and twill are completely orthogonal to  
> unittest - unittest does not have *any* web testing capabilities but  
> these tools can be driven from unittest to do web testing. We drive  
> Selenium from our unittest framework for example.
>> and buildbot or
>>   
> buildbot is also orthogonal - it is a test running framework that works  
> fine with unittest.
>
>> py.test better for integration testing.  
>
> :-)
>
> I'm *very* happy that py.test and nose are pushing forward the state of  
> the art in Python testing. I don't like some aspects of the patterns  
> they make possible myself (moving away from object oriented test  
> organisation), but diversity is a good thing. I don't see unittest as  
> fundamentally flawed.
>
> What is weak about unittest is the interaction of the various classes -  
> results, loaders, runners and the like which make it hard to provide  
> generic extensions of unittest. We have managed to extend unittest with  
> a custom runner that pushes results to a database - but combining that  
> with someone else's custom runner would be hard.
>
> The py.test and nose plugins are something I'd like to look at for  
> unittest in the future.
>
>> And the likes of
>> Windmill, twill or figleaf help with certain kinds of testing
>> that do not depend on JUnit style setups.   
>
> They don't depend on it but nor are they replacements for it.
>> I think using Python functions for expressing tests is great but that 
>> trying to fit setup/teardown of multiple test resources, databases, 
>> processes, network servers etc. into the JUnit setUp/tearDown model too 
>> limiting.   
>
> I agree that a class or module level setUp / tearDown for expensive  
> fixtures are something that could be introduced into unittest. They do  
> violate test isolation though and can encourage the writing of bad  
> tests. Definitely something to think about for the future, but not an  
> itch I feel the need to scratch at the moment.
>
>
>> Maybe a bit of personal experience: I have done and seen some
>> scarily large and involved setUp/tearDown functions.  With
>> inheritance hierarchies and copy-pasted-plus-slight-modified setup code 
>> fragments.  With self.__dict__ being more a global-ish name space with 
>> multiple places that write to it
>> and read from it.  Where for writing a test function I had to keep the 
>> setup scenario of my particular test class
>> in mind.  Where tests functions became hard to move across classes or 
>> files. 
>>
>> The worst, however, is that when i wanted to change the
>> way application or components are initialized or bootstrapped
>> and how test resources are setup i had to change a lot of
>> places in tests.  Suddenly tests didn't help refactoring, but rather 
>> hindered it because even a simple change could lead to many changes in 
>> test files and only few changes to app code.  
>>
>>   
>
> The writing of bad tests is possible in any framework - those sound like  
> very bad tests to me. We have several different TestCase base classes  
> for different types of tests:
>
> * FunctionalTestCase
> * ServerTestCase
> * GUIThreadTestCase
> * InstallerTestCase
>
> and so on. Each of these have their own setUp and tearDown and some of  
> them extend each other - FunctionalTestCase extends GUIThreadTestCase  
> for example. We find this object oriented pattern easy to understand and  
> extend. As I said, I see weaknesses in unittest elsewhere, but I don't  
> think the problems you mention are something that an alternative  
> framework would particularly make less likely...
>
> Michael Foord
>
> -- 
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog
>
>

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 