[TIP] unittest & TDD

[Michael Foord]

Collin Winter wrote:
> On 3/11/07, jason pellerin <jpellerin at gmail.com> wrote:
>   
>> On 3/11/07, Titus Brown <titus at caltech.edu> wrote:
>>     
>>> On Sun, Mar 11, 2007 at 08:35:32AM -0500, Atul Varma wrote:
>>> -> On 3/4/07, Bob Ippolito <bob at redivi.com> wrote:
>>> -> >I agree that unittest is straightforward, but it's annoying (on its
>>> -> >own). Too much unnecessary cruft.
>>> ->
>>> -> I agree.  Out of curiosity, is there any movement towards making
>>> -> py.test or nose a part of Python's standard library?  The main reason
>>> -> my company used unittest is, quite honestly, because we didn't look at
>>> -> any other testing options when we started writing our unit tests,
>>> -> assuming that the testing framework that shipped with Python was just
>>> -> "the one way to do it" (a la Python's motto).  We weren't huge fans of
>>> -> its complexity (or rather, the boilerplate cruft caused by said
>>> -> complexity), but we dealt with it.  The fact that unittest was pretty
>>> -> well documented helped a lot, too, but as soon as I saw py.test I had
>>> -> wished I'd known about it before I started writing unit tests.
>>>
>>> hi, Atul,
>>>
>>> IMO this is partly a "release schedule" problem and partly a community
>>> problem.
>>>
>>> First, I don't get the impression that either nose or py.test are at a
>>> stable resting point.  I could be wrong.
>>>       
>> That's certainly true of nose. It's nowhere near a stdlib level of
>> stability. Plus, I hope the new unittest in py3k is going to be much
>> different (and much better IMO)*, so I think the right time to talk
>> about integrating new test loading or running into stdlib will be more
>> like a couple of years from now.
>>
>> * In case anyone hasn't seen it, Collin Winter's draft design for
>> unittest in 3k is well worth checking out:
>> http://oakwinter.com/code/a-new-unittest/. I think it will really help
>> to untangle the various concerns of test loading, running and
>> reporting.
>>     
>
> Thanks for the vote of confidence, Jason : )
>
> Python 3000 will happen sooner than you think. Our goal is to get the
> first alpha out sometime this summer, with 3.0 to follow in the first
> half of 2008. With that in mind, I think the time to talk about new
> test loading and loading strategies is *now*, not a couple of years
> from now. The more input I get now, the less chance there is of me
> having to rewrite unittest *again* for Python 4K.
>
> I'm not so much looking for code as I am for ideas. The more ideas
> about test discovery/running/reporting/etc I have in hand, the more
> data points I have to work from when figuring out how this thing
> should operate. A sampling of the use-cases/extensions I'm working
> with now:
>
> * supporting refcount checking around each test.
> * marking certain tests as TODO.
> * skipping certain tests based on boolean criteria.
> * formatting test results a la the current unittest.
> * writing test results to an XML file.
> * emitting TAP (Perl's Test Anything Protocol) for a test run.
> * making sure you ran the expected number of tests.
>
>   
I've personally never understood the grievances against unittest. We've 
built a very large test framework on top of it, with some fairly 
advanced functionality, and it largely felt very natural to do so.

We've obviously not hit the specific areas that caused problems for you.

Features we've added include :

* Pushing test results into a database *whilst running* (so you can see 
the state of any build - including tracebacks - whilst it is running)
* A background error checking thread that looks for .NET crash dialogs 
(unfortunately .NET has several types of crash dialogs, not all of which 
will cause tests to fail)
* Some tests run on the GUI thread, others on a separate thread

etc, etc

The *main* advantage of unittest is that it is very easy to start with. 
Writing and running individual tests is trivially easy.

So long as any new framework is as easy to extend and as simple to use. 
It is hard to remember how to create a suite and add multiple test 
classes to it, but not very difficult to do.

FWIW I like the fact that tests start with 'test', and would find a 
decorator more annoying.

I like the idea of being able to run tests in a random order, we've 
sometimes had order dependent bugs.

The *most* annoying thing about unittest is that when you use 
'assertEquals', it doesn't show you what the actual value was (just that 
it wasn't equal to the expected value). Whenever I use unittest I 
ovveride 'assertEquals'.

All the best,


Fuzzyman
http://www.voidspace.org.uk/python/articles.shtml