[TIP] Doctest or unitest?

[Jim Fulton]

On Mar 5, 2007, at 7:42 PM, Andrew Bennetts wrote:

> Jim Fulton wrote:
> [...]
>>>
>>> - Certain operations are simply more awkward in a doctest.   
>>> Reusing  common
>>> set up and tear down is harder (you can't just subclass or  
>>> import  from a
>>> doctest), so people tend to just keep adding to the existing  
>>> doctest file
>>> rather than making a new file when they should.  Again the result  
>>> is long,
>>> rambling doctest files that are both poor tests and poor  
>>> documentation.
>>
>> This certainly doesn't need to be the case.  I usually do test setup
>> in Python.  I use the unittest integration of doctests and you can
>> pass set-up and tear-down functions to this interface.  I generally
>> avoid setup in the text unless it adds to the story.
>
> But this is more awkward than doing so in a TestCase class.  You  
> have to edit
> multiple files to arrange this.  The reader can't simply scroll  
> within the file
> to see what the assumed context (i.e. setUp) is for the code  
> examples to
> actually work.

In practice, I've seen a similar problem with TestCase based tests.   
You end up searching a bunch of base classes, trying to piece  
together what the context was from lots of different places.  I find  
that TestCase tests encourage abstraction that tends to be very harmful.

>   If I want to base a new test off an existing one, I have to
> randomly grep around until I find where the DocTestSuite is  
> constructed and
> chase where the setUp is defined, rather than knowing that I can  
> simply extract
> the setUp method right in front of me into a common base class.

I guess that this depends on how you construct your tests.  Our  
packages have a single tests module that defines or sets up all of  
the tests in that package. If you find a text file, you know to look  
for a tests file in the same package.  (Note that this is *current  
practice.) I haven't found finding the setup to be that big a deal.

Note that I find that there are two kinds of setup code, code that is  
interesting to the reader, and code that isn't.  Obviously, the  
former should be included in the documentation itself.  We've  
developed an extension to doctest that allows us to put some setup  
and edge case code that *might* be interesting to the user in ReST  
footnotes.  This lets us deal with borderline cases and also lets us  
reset the world conveniently many times in a doctest, since the  
footnoted code is run for every footnote reference. (We really need  
to make this extension more widely available.)

> As a result, I've seen people simply choose to extend an existing  
> doctest file
> because figuring out how to make a new one was too much hassle.

Yup

> This isn't a huge problem, but I do think it's fair to say that  
> doctests are
> a bit more awkward to work with in this respect than other tests.

I understand and appreciate your point, however, I prefer the  
situation in doctest for myself because it is so restricted and  
discourages the sort of abstractions encouraged by TestCase that I  
find usually makes TestCase tests hard to follow.

>> I haven't found this to be an issue myself.  This could just be a
>> matter of usage patterns. WRT line numbers, our test failure outputs
>> do in fact produce correct line numbers. I don't know why your's
>> aren't. Also, doctests do have names that can be used for test
>> selection.  Of course, you can only select the entire text body.
>
> With regards to line numbers, I think maybe that bug has been fixed  
> now (maybe
> in python 2.4?).

Ah, probably.  The file-based doctests in 2.3 might have had that  
problem.  While we still do development with Python 2.3, we use the  
Python 2.4 based doctests in 2.3. This led us to make a copy of the  
2.4 doctest module in zope.testing, which now means we've forled  
doctest. :(  We need to fix this. Having packages in the standard  
library is a pain.  It would have been easier to make the doctest  
fixes widely available sooner if doctest was distributed separately.  
Bit that's a different controversy. :)

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org