[TIP] Declarative style acceptance tests (was: Using doctest for functional tests / user stories)

Sean Fisk sean at seanfisk.com
Mon Jan 6 14:34:46 PST 2014


On Mon, Jan 6, 2014 at 3:53 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 6 January 2014 20:12, Ben Finney <ben+python at benfinney.id.au> wrote:
> > Paul Moore <p.f.moore at gmail.com> writes:
> >
> >> Is doctest considered a bad idea for this type of high-level narrative
> >> style of functional test (or spec)?
> >
> > Doctest is best used for testing examples that already occur in
> > documentation, such as API references or tutorials.
>
> Thanks. I was aware that was the intended use for doctest, and I had
> also seen plenty of comments to the effect that doctest is "bad" for
> general testing. But I wasn't able to find much in the way of clear
> explanations of *why* it is bad. Obviously it depends on what sort of
> tests you're trying to do - to me, it's clearly a bad fit for unit
> tests, but I'm less clear as to why it's inappropriate for functional
> tests (you refer to "acceptance" tests - forgive my ignorance, I'm not
> entirely clear what the difference is here).
>
> > For constructing rigorous acceptance tests, Doctest is a poor choice.
>
> Can you clarify why? It may be that the reason is that what you're
> referring to as acceptance tests is not what I'm trying to create (see
> below).

Hi Paul,

I would tend to agree with Ben in that doctest is not appropriate for
constructing “real” tests.

Since doctest interactive sessions are just Python code, they could
technically be used to do anything that would be done in normal Python
code. But they can be annoying to format because most editors don’t
understand them, and tools like flake8 won’t check them. In addition, most
tests I write require some sort of setup and benefit from test isolation,
and doctest doesn’t do too well in those areas.

I think that the reason you don’t see doctests used in this way is one of
intention. I view doctest as a convenient way to verify that my
documentation examples are still valid, *not* as a way to verify that my
program is running correctly. Therefore, I don’t worry about things like
coverage or mocking when dealing with doctest. I just want to make sure my
documentation is still correct.

>
> >> Or is the whole idea of structuring the user stories as testable
> >> documents considered bad practice? I'm finding it very difficult to
> >> formulate good functional tests in the unittest style, but I don't
> >> want to switch to doctest if it's going to cause me problems as my
> >> development gets more complex.
> >
> > You're right, unit tests are not suited for writing acceptance tests.
>
Unit tests are not suited for writing acceptance tests, but that doesn’t
mean a so-called "unit testing" framework (such as
pytest<http://pytest.org/latest/>)
isn’t suitable for writing acceptance tests.

>
> Is there a good reference for the differences between unit,
> functional, integration and acceptance tests? (And any other similar
> terms I might not have thought of.) "Testing Web Apps in Python" talks
> about the initial functional tests as being essentially step-by-step
> "the user does X and expects to see Y" descriptions - what I'm
> thinking of as "user stories".

The main difference *that I see* between these types of testing is this:

   - Unit testing is typically white box testing.
   - Functional, acceptance, system, and integration testing is typically
   black box testing.

For a small command-line program, I wouldn’t get hung up too much on the
terms. I haven’t yet worked on a program large enough to employ all these
types of testing, but they may come into play for very large codebases.

>
> > There are better options. It's best to have acceptance tests in as close
> > a form to statements the customer can understand; this means declarative
> > assertions in something close to natural English. These declarative
> > assertions can then be parsed and tested by executable test code.
>
> In the case of my application, the end user is basically myself. I am
> writing a command line utility. In actual fact, the CLI is essentially
> likely to be a thin wrapper around a library API, and my real focus at
> this point is on getting the API correct. Something like
>
>     db = myapp.DB()
>     db.create_schema()
>     datasource = Source("web app")
>     for batch in datasource.load(batchsize=100):
>         db.load(batch)
>
> (At the command line, that'd be something like "myapp create_schema"
> and "myapp load --batchsize=100 --source='web app'" - hence my comment
> that the CLI is a thin wrapper).
>
> Writing a "narrative" like that gives me a good idea of how I want the
> API to work, and I was asking my question because that feels so close
> to a doctest.
>
> If I hadn't seen a number of posts saying "doctest shouldn't be used
> for XXX tests" (unit, functional,...) I would probably just have gone
> ahead and done things that way. But I'd like to understand *why* I'd
> end up in trouble taking that approach (preferably without having to
> take that route and fail first ;-))

I wouldn’t necessarily say that you’d fail; you just might find out that
doctest isn’t well-suited to writing those kinds of tests (for the
aforementioned reasons). If most of your users will be technical, then I
don’t see a problem with writing technical tests.

If it were me, I’d probably just use pytest for both unit and functional
tests. I’m not trying to push it, but I haven’t found much that it
*can’t*do. If you would like an example of some unit and functional
tests, I’d be
happy to provide them. Hope this helps!

>
> >> If anyone has any thoughts or suggestions, or better alternatives I'd
> >> be very grateful.
> >
> > One that I recommend is Behave <URL:http://pypi.python.org/pypi/behave>
> > which reads acceptance tests written in a declarative, formalised, but
> > plain-to-read English style; and then the programmer writes test
> > functions which define what the terms in those declarations mean. Behave
> > then tests the teclarations using your test code, and reports the
> > result.
>
> Behave looks interesting. But the PyPI page says BDD "encourages
> collaboration between developers, QA and non-technical or business
> participants". In my case there's no non-technical participants (heck,
> there's nobody but me, right now) so is BDD really the right tool
> here? Also, I'm only just starting with TDD, I'm pretty sure I'll get
> confused adding BDD into the mix at the same time! But I'll definitely
> take a look. It may well be that by forcing myself to think of my
> requirements in the Behave style, I'll get a clearer picture of the
> design I'm trying to achieve.
>
> Thanks for the suggestion,
> Paul
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20140106/a45d055d/attachment.htm>


More information about the testing-in-python mailing list