[TIP] On using doctests all the time WAS: Python Testing book review

[Andrew Bennetts]

Just quickly, because much of this discussion has been had here before:

Olemis Lang wrote:
[...]
> > I think automatically tested documentation is a wonderful idea.
> 
> Cool. IMO automatically tested documentation is not just for writing
> books, but also to document the code ;o)

Yes, definitely. 

(But tested documentation of the code is unlikely to be a good replacement for a
typical test suite, as readers of documentation generally don't care about all
the corner cases your test suite should cover.)

> > This is an orthogonal concern to whether a good way to write precise and
> > maintainable assertions is by turning every single outcome to assert
> > into a string and then comparing it to another string.
> 
> Good point . I think that, as long as True be printed as True and
> False printed as False there's almost no difference between writing
> doctests and writing unittests or other kinds of assertions . If you

Using purely True/False values in doctests you get much less informative
failures, though.  e.g. compare the failures you'd get from assertEqual(42,
something) with:

    >>> something == 42
    True

This is especially vexing if your test suite is slow or the failure is
intermittent.

[...]
> > I think that's a
> > terrible idea.
> 
> ... to rely on string matching using non-standard features ? IMO yes,
> but the same happens when writing TestCase(s) to perform string
> matching . So I think that's not an drawback of the framework .

Actually, it's not matching “non-standard features” that is the problem
exactly.  The u'' prefix for unicode strings is a completely standard
feature of Python 2.x, after all.  The problem is overly-specific tests,
tests that are asserting unimportant details rather than just precisely
the things the test is intended to assert.  So in the case of MvL's
problem on python-dev, what the test wanted to assert was that the
string equalled a particular value and that its type was unicode.
Instead what was tested was that repr(that_string) equalled another
string.  So even though mostly likely the code being tested works
equally well when repr of unicode strings is different, the test suite
reporting failures.  The test suite is overly specific.

See also
<http://xunitpatterns.com/Fragile%20Test.html#Sensitive%20Equality>.

> So I use doctest to write my tests 'cause
> 
>   - They are quite readable
>   - ... so even users can read them and use that as a
>      reference, tutorial, ...

In my experience, tests make poor end-user docs, although it depends a
little on the sort of user (users of your API vs. of your command line
etc).  I suppose it's better than no docs at all

Again, tested docs are great, but tests and documentation have different
audiences.

>   - ... and they can be part of the testing efforts and
>      write doctests for you (e.g. user find a bug, you invite him
>      to write a snippet illustrating what he was doing
>      -rather than writing a unittest test case or alike-, and voilà !
>      there you have a new test scenario ;o)

It's nice to make it as easy as possible for potential contributors to
contribute tests.  I think in the long run I'd end up refactoring most
of them to be regular unit tests; new contributors often don't have
enough time and knownledge to make a very precise and focussed test, but
they can often turn what they did into something that at least
reproduces the failure.  That's a great start, and better than having no
test, but if your whole test suite was written like that it would be
slow and imprecise and as much a burden to on-going maintenance as a
help.

>   - I don't have time to write both docs and tests
>     (especially when doing the same thing ;o)

If they are actually the same, perhaps.  I know how hard it is to find
time to write docs, but it is worth it for the users.  In fact, it is
often worth writing the “same thing” multiple times in different
documents, targetted to different audiences: one or more tutorials on
particular topics, a comprehensive reference, a glossary, a quickstart
guide for users familiar with a competing system, etc.

The primary audience for tests is fellow developers attempting to work
on the software.  The audience for a document might also be a fellow
developer, but it is most likely to be a different sort of person.  If
you can find the time, it is usually worth tailoring your writing to the
audience as much as possible.

-Andrew.