[TIP] [Fwd: Re: suitability of unittest.py for large-scale tests]

Michael Foord fuzzyman at voidspace.org.uk
Sun Apr 5 09:34:02 PDT 2009

Holger wanted this sent to the list as proof that we're all still 
friends. :-)

-------- Original Message --------
Subject: 	Re: suitability of unittest.py for large-scale tests
Date: 	Sun, 5 Apr 2009 18:30:00 +0200
From: 	holger krekel <holger at merlinux.eu>
To: 	Michael Foord <fuzzyman at voidspace.org.uk>
References: 	<49D6BAC9.1060300 at nedbatchelder.com> 
<20090404022702.GA17605 at idyll.org> <20090404071836.GR8296 at trillke.net> 
<49D74DF1.102 at voidspace.org.uk> <20090404130926.GS8296 at trillke.net> 
<49D75F50.4090306 at voidspace.org.uk> <20090404155440.GT8296 at trillke.net> 
<49D78850.4090500 at voidspace.org.uk> <20090404173445.GV8296 at trillke.net> 
<49D79DFA.40400 at voidspace.org.uk>

Hi Michael, 

On Sat, Apr 04, 2009 at 18:50 +0100, Michael Foord wrote:
> Hey Holger,
> It's been a fun and useful discussion - thanks.

yip, i see it the same way!  If you want to send this closure 
comment also to the list it could maybe provide an example that
not all controversies need to end in flames :)

> I'm not really involved in pushing testing forwards in the same way as  
> you are - more trying to make unittest better at what it already does.  
> Once I've cleared my backlog now that the book is done I'd like to get  
> more onto the cutting edge. I have no specific plans though and no  
> intention of making any in a hurry.

I definitely want to work towards a more general test
running and reporting system - and that includes 
running it on IronPython.  So maybe we have some
co-operation area there :) 


> Michael
> holger krekel wrote:
>> Hi Michael, 
>> thanks for your mail.  I think we agree that there are a
>> lot of good test tools, methods and libs out which are mostly  
>> orthogonal to unittest.py.  I understand that unittest.py can
>> be extended through subclasses in order to deal with all kinds
>> of testing.  And that you and others have successfully done
>> so.  And that you want to work on improving unittest. 
>> I am glad to see that you are happily hacking on unittest.  In fact, i 
>> may well want to integrate the results of your work
>> into a py.test release so that people can use the new
>> mechanisms rather sooner and cross-python version. 
>> My current goal is to have all kinds of test libs and test types 
>> available at a fingertipp.  So I am currently hacking
>> on tools to collect, run and debug all kinds of tests, up to eventually
>> integrating with actual deployment.  I eventually see deployment as 
>> another kind of test run where i want to have helpful information when 
>> things go wrong :)  Much of this certainly is, as you point out,  
>> orthogonal to what unittest.py tries to provide.  
>> For me it's fine to leave it at that for now.  cheers and have fun,  
>> holger
>> Sat, Apr 04, 2009 at 17:18 +0100, Michael Foord wrote:
>>> holger krekel wrote:
>>>> On Sat, Apr 04, 2009 at 14:23 +0100, Michael Foord wrote:
>>>>> holger krekel wrote:
>>>>>> [snip...]
>>>>>> I think others did good postings and i agree (and also mailed
>>>>>> so) that the current addCleanup suggestions make some sense to me.
>>>>>> However, in Python voting style i'd say "-0" on it. 
>>>>>> Why? I tried to give background for my views that 
>>>>>> 1) unittest.py is maybe not the right place to drive larger 
>>>>>> scale  tests.                 
>>>>> Do you mean that you don't think unittest is suitable for testing 
>>>>> of  large applications or large systems?
>>>>> If so then could you elaborate? Of all the Python  test 
>>>>> frameworks it is  probably the most proven in this area of 
>>>>> course.
>>>> do you mean unittest.py is the most proven in the area
>>>> of large scale tests?  
>>> I would think so, although I would have a hard time proving it. By   
>>> virtue of its age and inclusion in the standard library it is almost  
>>> certainly the most widely used
>>> I also have some experience of using it with large test systems. At   
>>> Resolver One we have >130 000 lines of test code built on unittest 
>>> that  do unit testing, functional testing and web testing.
>>>> I regard many test frameworks and methods fit for purposes that  
>>>> unittest.py does not really cover. texttest, selenium or doctests 
>>>> are often better suited for
>>>> acceptance or certain functional testing.        
>>> Personally I find doctest fantastic for testing documentation 
>>> examples  but an awful general purpose testing tool. I realise that 
>>> others disagree.
>>> Tools like selenium, windmill and twill are completely orthogonal to  
>>> unittest - unittest does not have *any* web testing capabilities but  
>>> these tools can be driven from unittest to do web testing. We drive   
>>> Selenium from our unittest framework for example.
>>>> and buildbot or
>>> buildbot is also orthogonal - it is a test running framework that 
>>> works  fine with unittest.
>>>> py.test better for integration testing.        
>>> :-)
>>> I'm *very* happy that py.test and nose are pushing forward the state 
>>> of  the art in Python testing. I don't like some aspects of the 
>>> patterns  they make possible myself (moving away from object oriented 
>>> test  organisation), but diversity is a good thing. I don't see 
>>> unittest as  fundamentally flawed.
>>> What is weak about unittest is the interaction of the various classes 
>>> -  results, loaders, runners and the like which make it hard to 
>>> provide  generic extensions of unittest. We have managed to extend 
>>> unittest with  a custom runner that pushes results to a database - 
>>> but combining that  with someone else's custom runner would be hard.
>>> The py.test and nose plugins are something I'd like to look at for   
>>> unittest in the future.
>>>> And the likes of
>>>> Windmill, twill or figleaf help with certain kinds of testing
>>>> that do not depend on JUnit style setups.         
>>> They don't depend on it but nor are they replacements for it.
>>>> I think using Python functions for expressing tests is great but 
>>>> that trying to fit setup/teardown of multiple test resources, 
>>>> databases, processes, network servers etc. into the JUnit 
>>>> setUp/tearDown model too limiting.         
>>> I agree that a class or module level setUp / tearDown for expensive   
>>> fixtures are something that could be introduced into unittest. They 
>>> do  violate test isolation though and can encourage the writing of 
>>> bad  tests. Definitely something to think about for the future, but 
>>> not an  itch I feel the need to scratch at the moment.
>>>> Maybe a bit of personal experience: I have done and seen some
>>>> scarily large and involved setUp/tearDown functions.  With
>>>> inheritance hierarchies and copy-pasted-plus-slight-modified setup 
>>>> code fragments.  With self.__dict__ being more a global-ish name 
>>>> space with multiple places that write to it
>>>> and read from it.  Where for writing a test function I had to keep 
>>>> the setup scenario of my particular test class
>>>> in mind.  Where tests functions became hard to move across classes 
>>>> or files. 
>>>> The worst, however, is that when i wanted to change the
>>>> way application or components are initialized or bootstrapped
>>>> and how test resources are setup i had to change a lot of
>>>> places in tests.  Suddenly tests didn't help refactoring, but 
>>>> rather hindered it because even a simple change could lead to many 
>>>> changes in test files and only few changes to app code.  
>>> The writing of bad tests is possible in any framework - those sound 
>>> like  very bad tests to me. We have several different TestCase base 
>>> classes  for different types of tests:
>>> * FunctionalTestCase
>>> * ServerTestCase
>>> * GUIThreadTestCase
>>> * InstallerTestCase
>>> and so on. Each of these have their own setUp and tearDown and some 
>>> of  them extend each other - FunctionalTestCase extends 
>>> GUIThreadTestCase  for example. We find this object oriented pattern 
>>> easy to understand and  extend. As I said, I see weaknesses in 
>>> unittest elsewhere, but I don't  think the problems you mention are 
>>> something that an alternative  framework would particularly make less 
>>> likely...
>>> Michael Foord
>>> -- 
>>> http://www.ironpythoninaction.com/
>>> http://www.voidspace.org.uk/blog
> -- 
> http://www.ironpythoninaction.com/
> http://www.voidspace.org.uk/blog

Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu 


More information about the testing-in-python mailing list