[TIP] Doctest or unitest?
jim at zope.com
Wed Mar 7 01:45:15 PST 2007
On Mar 5, 2007, at 7:42 PM, Andrew Bennetts wrote:
> Jim Fulton wrote:
>>> - Certain operations are simply more awkward in a doctest.
>>> Reusing common
>>> set up and tear down is harder (you can't just subclass or
>>> import from a
>>> doctest), so people tend to just keep adding to the existing
>>> doctest file
>>> rather than making a new file when they should. Again the result
>>> is long,
>>> rambling doctest files that are both poor tests and poor
>> This certainly doesn't need to be the case. I usually do test setup
>> in Python. I use the unittest integration of doctests and you can
>> pass set-up and tear-down functions to this interface. I generally
>> avoid setup in the text unless it adds to the story.
> But this is more awkward than doing so in a TestCase class. You
> have to edit
> multiple files to arrange this. The reader can't simply scroll
> within the file
> to see what the assumed context (i.e. setUp) is for the code
> examples to
> actually work.
In practice, I've seen a similar problem with TestCase based tests.
You end up searching a bunch of base classes, trying to piece
together what the context was from lots of different places. I find
that TestCase tests encourage abstraction that tends to be very harmful.
> If I want to base a new test off an existing one, I have to
> randomly grep around until I find where the DocTestSuite is
> constructed and
> chase where the setUp is defined, rather than knowing that I can
> simply extract
> the setUp method right in front of me into a common base class.
I guess that this depends on how you construct your tests. Our
packages have a single tests module that defines or sets up all of
the tests in that package. If you find a text file, you know to look
for a tests file in the same package. (Note that this is *current
practice.) I haven't found finding the setup to be that big a deal.
Note that I find that there are two kinds of setup code, code that is
interesting to the reader, and code that isn't. Obviously, the
former should be included in the documentation itself. We've
developed an extension to doctest that allows us to put some setup
and edge case code that *might* be interesting to the user in ReST
footnotes. This lets us deal with borderline cases and also lets us
reset the world conveniently many times in a doctest, since the
footnoted code is run for every footnote reference. (We really need
to make this extension more widely available.)
> As a result, I've seen people simply choose to extend an existing
> doctest file
> because figuring out how to make a new one was too much hassle.
> This isn't a huge problem, but I do think it's fair to say that
> doctests are
> a bit more awkward to work with in this respect than other tests.
I understand and appreciate your point, however, I prefer the
situation in doctest for myself because it is so restricted and
discourages the sort of abstractions encouraged by TestCase that I
find usually makes TestCase tests hard to follow.
>> I haven't found this to be an issue myself. This could just be a
>> matter of usage patterns. WRT line numbers, our test failure outputs
>> do in fact produce correct line numbers. I don't know why your's
>> aren't. Also, doctests do have names that can be used for test
>> selection. Of course, you can only select the entire text body.
> With regards to line numbers, I think maybe that bug has been fixed
> now (maybe
> in python 2.4?).
Ah, probably. The file-based doctests in 2.3 might have had that
problem. While we still do development with Python 2.3, we use the
Python 2.4 based doctests in 2.3. This led us to make a copy of the
2.4 doctest module in zope.testing, which now means we've forled
doctest. :( We need to fix this. Having packages in the standard
library is a pain. It would have been easier to make the doctest
fixes widely available sooner if doctest was distributed separately.
Bit that's a different controversy. :)
Jim Fulton mailto:jim at zope.com Python Powered!
CTO (540) 361-1714 http://www.python.org
Zope Corporation http://www.zope.com http://www.zope.org
More information about the testing-in-python