[TIP] Declarative style acceptance tests

Ben Finney ben+python at benfinney.id.au
Mon Jan 6 18:51:21 PST 2014


Paul Moore <p.f.moore at gmail.com> writes:

> On 6 January 2014 20:12, Ben Finney <ben+python at benfinney.id.au> wrote:
> > For constructing rigorous acceptance tests, Doctest is a poor choice.
>
> Can you clarify why?

In brief: Doctest is for testing your *documentation*, not for testing
your code. The two purposes are in conflict: Good instances of
documentation are not necessarily good instances of rigorous tests.

A detailed (and somewhat more confrontationally-worded) explanation at
<URL:http://www.somethinkodd.com/oddthinking/2010/01/11/pythons-doctest-considered-harmful/>.
Unlike Julian, I wouldn't go so far as to say “is harmful”; rather,
Doctest is a good tool for testing documentation, and not a good tool
for anything else.

> Writing a "narrative" like that gives me a good idea of how I want the
> API to work, and I was asking my question because that feels so close
> to a doctest.

If you're testing an API, you're not writing acceptance (nor
“functional”) tests. You should be writing unit tests, since you're
testing the behaviour of discrete functions.

On the other hand, if you're wanting to test the whole behaviour of the
system with a user story, you're no longer testing the API and hence
unit tests are not what you're writing.

> If I hadn't seen a number of posts saying "doctest shouldn't be used
> for XXX tests" (unit, functional,...) I would probably just have gone
> ahead and done things that way. But I'd like to understand *why* I'd
> end up in trouble taking that approach (preferably without having to
> take that route and fail first ;-))

Have a look at typical documentation, and the code examples in there.
They test mostly the “happy path”, and rarely test the limits of the
system for failure. They're meant to show the user how the API should be
used.

The code examples aren't testing the whole system – and if they were,
Doctest makes it difficult to do the complex set-up needed. The code
examples aren't meant to be rigorously exhaustive – and if they were,
Doctest makes it difficult to be precise and/or isolate parts properly.

None of this is a fault in Doctest: it is very good for what it's
designed for, testing documentation examples.

> In my case there's no non-technical participants (heck, there's nobody
> but me, right now) so is BDD really the right tool here?

If you want to test the whole system's behaviour, I don't think
‘unittest’ is a clumsy tool, and Doctest certainly won't help.

For acceptance testing, you need to put the whole system in a harness
that allows you to test the whole-system behaviour, and acceptance-test
tools like Behave (and Cucumber, and FITnesse, etc.) are a good tool.

There just isn't a good acceptance testing harness in the standard
library (and I'm not even convinced it belongs there), which IMO is why
people try too hard to press ‘unittest’ and ‘doctest’ into service where
they don't fit very well.

> Also, I'm only just starting with TDD, I'm pretty sure I'll get
> confused adding BDD into the mix at the same time!

It's good to learn at your own pace. But the two aren't in conflict: you
can learn BDD and TDD at the same time, because they're pretty much the
same thing at different levels of testing, and are entirely
complementary techniques.

-- 
 \         “A child of five could understand this. Fetch me a child of |
  `\                                              five.” —Groucho Marx |
_o__)                                                                  |
Ben Finney




More information about the testing-in-python mailing list