[TIP] suitability of unittest.py for large-scale tests
fuzzyman at voidspace.org.uk
Sat Apr 4 09:18:24 PDT 2009
holger krekel wrote:
> On Sat, Apr 04, 2009 at 14:23 +0100, Michael Foord wrote:
>> holger krekel wrote:
>>> I think others did good postings and i agree (and also mailed
>>> so) that the current addCleanup suggestions make some sense to me.
>>> However, in Python voting style i'd say "-0" on it.
>>> Why? I tried to give background for my views that
>>> 1) unittest.py is maybe not the right place to drive larger scale
>> Do you mean that you don't think unittest is suitable for testing of
>> large applications or large systems?
>> If so then could you elaborate? Of all the Python test frameworks it is
>> probably the most proven in this area of course.
> do you mean unittest.py is the most proven in the area
> of large scale tests?
I would think so, although I would have a hard time proving it. By
virtue of its age and inclusion in the standard library it is almost
certainly the most widely used
I also have some experience of using it with large test systems. At
Resolver One we have >130 000 lines of test code built on unittest that
do unit testing, functional testing and web testing.
> I regard many test frameworks and methods fit for
> purposes that unittest.py does not really cover.
> texttest, selenium or doctests are often better suited for
> acceptance or certain functional testing.
Personally I find doctest fantastic for testing documentation examples
but an awful general purpose testing tool. I realise that others disagree.
Tools like selenium, windmill and twill are completely orthogonal to
unittest - unittest does not have *any* web testing capabilities but
these tools can be driven from unittest to do web testing. We drive
Selenium from our unittest framework for example.
> and buildbot or
buildbot is also orthogonal - it is a test running framework that works
fine with unittest.
> py.test better for integration testing.
I'm *very* happy that py.test and nose are pushing forward the state of
the art in Python testing. I don't like some aspects of the patterns
they make possible myself (moving away from object oriented test
organisation), but diversity is a good thing. I don't see unittest as
What is weak about unittest is the interaction of the various classes -
results, loaders, runners and the like which make it hard to provide
generic extensions of unittest. We have managed to extend unittest with
a custom runner that pushes results to a database - but combining that
with someone else's custom runner would be hard.
The py.test and nose plugins are something I'd like to look at for
unittest in the future.
> And the likes of
> Windmill, twill or figleaf help with certain kinds of testing
> that do not depend on JUnit style setups.
They don't depend on it but nor are they replacements for it.
> I think using Python functions for expressing tests is
> great but that trying to fit setup/teardown of multiple
> test resources, databases, processes, network servers etc.
> into the JUnit setUp/tearDown model too limiting.
I agree that a class or module level setUp / tearDown for expensive
fixtures are something that could be introduced into unittest. They do
violate test isolation though and can encourage the writing of bad
tests. Definitely something to think about for the future, but not an
itch I feel the need to scratch at the moment.
> Maybe a bit of personal experience: I have done and seen some
> scarily large and involved setUp/tearDown functions. With
> inheritance hierarchies and copy-pasted-plus-slight-modified
> setup code fragments. With self.__dict__ being more a
> global-ish name space with multiple places that write to it
> and read from it. Where for writing a test function I
> had to keep the setup scenario of my particular test class
> in mind. Where tests functions became hard to move across
> classes or files.
> The worst, however, is that when i wanted to change the
> way application or components are initialized or bootstrapped
> and how test resources are setup i had to change a lot of
> places in tests. Suddenly tests didn't help refactoring, but
> rather hindered it because even a simple change could lead to
> many changes in test files and only few changes to app code.
The writing of bad tests is possible in any framework - those sound like
very bad tests to me. We have several different TestCase base classes
for different types of tests:
and so on. Each of these have their own setUp and tearDown and some of
them extend each other - FunctionalTestCase extends GUIThreadTestCase
for example. We find this object oriented pattern easy to understand and
extend. As I said, I see weaknesses in unittest elsewhere, but I don't
think the problems you mention are something that an alternative
framework would particularly make less likely...
More information about the testing-in-python