[TIP] Doctest or unitest?

[Andrew Bennetts]

Jim Fulton wrote:
[...]
> potentially a lot harder.  When you read normal code, the intent of  
> the code is usually known.  With tests, the intent is far less  
> obvious.  I find that doctests tend to make test code more  
> understandable because:

I find even normal code can have extremely obscure intent.  The "how" is always
obvious (that's what code is), but the intent needs explicit care from the
author if it is to be communicated to the next person to look at the code.  Good
habits in naming of identifiers and the like obviously go some way to helping
here, so that taking this care becomes somewhat automatic, but it needs to be
taken.

> - They don't litter the code with superfluous testing framework calls,
> 
> - They make it easier to write prose that code.  Some people defend  
> unittest-style tests by telling me that it is easy to write comments  
> in unittest-based unit tests, but I find this rarely happens in  
> practice.
> 
> - doctests force you to go further out of your way to create  
> "helpful" abstractions that allow code sharing between tests and that  
> make reading tests much more difficult.
> 
> Of course, doctests don't prevent inscrutable tests.

Indeed :)

Not to disagree with your points in favour, but there are some aspects that can
make doctests less readable than plain unittest.TestCases:

- A doctest file tends to accumulate context, so that by the end of the file the
  reader can't easily determine which preceding examples are revelant and
  which aren't.  By stuffing several, often dozens, of tests into the one test,
  interdependencies aren't clear.  (The lack of isolation also harms the
  reliability/correctness of the tests, but my point here is that it harms
  readability too.)

- Exercising as many edge cases and error conditions as possible to get
  comprehensive test coverage detracts considerably from the clarity of the
  documentation that a doctest purports to be.  Screens and screens of "Show
  that frobnicator() does foo when x is 17... also when y is 18" don't make good
  documentation (and the intent of the test is just as unclear as if it were
  written in a TestCase... why do foo in that case?  I can see that it does, but
  why?).

- Certain operations are simply more awkward in a doctest.  Reusing common set
  up and tear down is harder (you can't just subclass or import from a doctest),
  so people tend to just keep adding to the existing doctest file rather than
  making a new file when they should.  Again the result is long, rambling
  doctest files that are both poor tests and poor documentation.

- It's harder to name and talk about a specific failing test case when your test
  case is example 27, about, oh, half-way down foo-feature.txt, compared to
  "FooTestCase.test_feature_with_all_knobs_turned_off".  This is frustrating
  when communicating with fellow developers about a test failure, and is
  exacerbated by the lack of correlation between reported line numbers in errors
  and actual line numbers in the file.  (This also mildly irritating when trying
  to run just the failing test for purposes of debugging and fixing the
  failure.)

I've seen all these problems occur.

Of course, it is still possible to write clear doctests.  

One of the nicest uses of doctests I've seen is page tests using
zope.testbrowser.  The narrative format of a doctest fits very neatly with the
thing being tested, which is a story about how a user interacts with a website.

The other good use is as testable *documentation*, with the emphasis on
documentation.  If a doctest file is primarily there to provide test coverage,
rather than explaining how to use something, then in my experience it's probably
in the wrong format.

I've seen plenty of inscrutable unittest.TestCase classes too (and py.test
functions...), of course.  Doctest or not, there's no substitute for good taste
:)

-Andrew.