[TIP] Doctest or unitest?

[Jim Fulton]

On Mar 5, 2007, at 8:53 AM, Andrew Bennetts wrote:
...

> - A doctest file tends to accumulate context, so that by the end of  
> the file the
>   reader can't easily determine which preceding examples are  
> revelant and
>   which aren't.  By stuffing several, often dozens, of tests into  
> the one test,
>   interdependencies aren't clear.  (The lack of isolation also  
> harms the
>   reliability/correctness of the tests, but my point here is that  
> it harms
>   readability too.)

Agreed.  This is definitely a danger.  This is, to some degree, a  
matter of knowing how to use the tool.  It is a good idea to prevent  
stories from getting too long.  If you're writing documentation, the  
emphasis needs to be on readability and often this will often require  
splitting the story up.

(I'm not persuaded by the argument that combining tests this way  
hurts reliability or correctness.  Certainly, there is some advantage  
to having independent tests from a convenience point of view.)

> - Exercising as many edge cases and error conditions as possible to  
> get
>   comprehensive test coverage detracts considerably from the  
> clarity of the
>   documentation that a doctest purports to be.

Right, that's why we avoid putting these in the documentation. We  
typically have documentation files that are tests and separate tests  
do deal with edge cases, and regressions.  These additional tests can  
be expressed any way one wants, although I still prefer the  
readability of doctests for these.  These are typically in separate  
test modules.

>   Screens and screens of "Show
>   that frobnicator() does foo when x is 17... also when y is 18"  
> don't make good
>   documentation (and the intent of the test is just as unclear as  
> if it were
>   written in a TestCase... why do foo in that case?  I can see that  
> it does, but
>   why?).

Agreed. Something FIT-based seems more appropriate in a case like  
this. In fact, I'd like to find a way to integrate FIT and doctests,  
so you can intersperse code examples and tables.

>
> - Certain operations are simply more awkward in a doctest.  Reusing  
> common set
>   up and tear down is harder (you can't just subclass or import  
> from a doctest),
>   so people tend to just keep adding to the existing doctest file  
> rather than
>   making a new file when they should.  Again the result is long,  
> rambling
>   doctest files that are both poor tests and poor documentation.

This certainly doesn't need to be the case.  I usually do test setup  
in Python.  I use the unittest integration of doctests and you can  
pass set-up and tear-down functions to this interface.  I generally  
avoid setup in the text unless it adds to the story.

> - It's harder to name and talk about a specific failing test case  
> when your test
>   case is example 27, about, oh, half-way down foo-feature.txt,  
> compared to
>   "FooTestCase.test_feature_with_all_knobs_turned_off".  This is  
> frustrating
>   when communicating with fellow developers about a test failure,  
> and is
>   exacerbated by the lack of correlation between reported line  
> numbers in errors
>   and actual line numbers in the file.  (This also mildly  
> irritating when trying
>   to run just the failing test for purposes of debugging and fixing  
> the
>   failure.)

I haven't found this to be an issue myself.  This could just be a  
matter of usage patterns. WRT line numbers, our test failure outputs  
do in fact produce correct line numbers. I don't know why your's  
aren't. Also, doctests do have names that can be used for test  
selection.  Of course, you can only select the entire text body.

> I've seen all these problems occur.

I've seen most of them.

Personally, I still vastly prefer doctests to unittest tests.

...

> I've seen plenty of inscrutable unittest.TestCase classes too (and  
> py.test
> functions...), of course.  Doctest or not, there's no substitute  
> for good taste
> :)

Agreed.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org