[TIP] unittest & TDD

[Collin Winter]

On 3/11/07, jason pellerin <jpellerin at gmail.com> wrote:
> On 3/11/07, Titus Brown <titus at caltech.edu> wrote:
> > On Sun, Mar 11, 2007 at 08:35:32AM -0500, Atul Varma wrote:
> > -> On 3/4/07, Bob Ippolito <bob at redivi.com> wrote:
> > -> >I agree that unittest is straightforward, but it's annoying (on its
> > -> >own). Too much unnecessary cruft.
> > ->
> > -> I agree.  Out of curiosity, is there any movement towards making
> > -> py.test or nose a part of Python's standard library?  The main reason
> > -> my company used unittest is, quite honestly, because we didn't look at
> > -> any other testing options when we started writing our unit tests,
> > -> assuming that the testing framework that shipped with Python was just
> > -> "the one way to do it" (a la Python's motto).  We weren't huge fans of
> > -> its complexity (or rather, the boilerplate cruft caused by said
> > -> complexity), but we dealt with it.  The fact that unittest was pretty
> > -> well documented helped a lot, too, but as soon as I saw py.test I had
> > -> wished I'd known about it before I started writing unit tests.
> >
> > hi, Atul,
> >
> > IMO this is partly a "release schedule" problem and partly a community
> > problem.
> >
> > First, I don't get the impression that either nose or py.test are at a
> > stable resting point.  I could be wrong.
>
> That's certainly true of nose. It's nowhere near a stdlib level of
> stability. Plus, I hope the new unittest in py3k is going to be much
> different (and much better IMO)*, so I think the right time to talk
> about integrating new test loading or running into stdlib will be more
> like a couple of years from now.
>
> * In case anyone hasn't seen it, Collin Winter's draft design for
> unittest in 3k is well worth checking out:
> http://oakwinter.com/code/a-new-unittest/. I think it will really help
> to untangle the various concerns of test loading, running and
> reporting.

Thanks for the vote of confidence, Jason : )

Python 3000 will happen sooner than you think. Our goal is to get the
first alpha out sometime this summer, with 3.0 to follow in the first
half of 2008. With that in mind, I think the time to talk about new
test loading and loading strategies is *now*, not a couple of years
from now. The more input I get now, the less chance there is of me
having to rewrite unittest *again* for Python 4K.

I'm not so much looking for code as I am for ideas. The more ideas
about test discovery/running/reporting/etc I have in hand, the more
data points I have to work from when figuring out how this thing
should operate. A sampling of the use-cases/extensions I'm working
with now:

* supporting refcount checking around each test.
* marking certain tests as TODO.
* skipping certain tests based on boolean criteria.
* formatting test results a la the current unittest.
* writing test results to an XML file.
* emitting TAP (Perl's Test Anything Protocol) for a test run.
* making sure you ran the expected number of tests.

I'm also investigating bringing several features from py.test into my
new design, specifically generative tests [1]; running tests in random
order; and traceback snipping/beautification.

Not all of these will make it into the core proposal for Python 3.0,
but by designing unittest to accommodate as many varied use cases as
possible, the more flexible I hope to make the internal framework.

So, let me have any ideas you may have, any grievances and grudges you
may hold against the existing unittest.

Thanks,
Collin Winter

[1] - http://codespeak.net/py/dist/test.html#generative-tests-yielding-more-tests