[TIP] Fwd: An OO API for doctest / unittest integration...
Olemis Lang
olemis at gmail.com
Thu Aug 14 08:25:56 PDT 2008
Hello... this is my first message I send to this list... so I'ld like
to introduce myself... briefly. I knew Python since about one year
ago... and I fell in love with it right away. I spend most of my time
developping some libraries for the FLiOOPS project [2]... and this
post is related to a simple test support module developped to ensure
the quality of our work in the aforementioned project.
As you can see below, I posted a similar message to python-dev [13],
but there was "general consensus" about using this list for discussing
these topics, so... let's be it.
Before submitting this message I read some threads about doctest and
unittest [11] in this list. When talking about the fact that "there
are some aspects that can
make doctests less readable than plain unittest.TestCases" Its author
(Andrew Bennetts I think...) said something like this:
> - A doctest file tends to accumulate context, so that by the end of the file the
> reader can't easily determine which preceding examples are revelant and
> which aren't. By stuffing several, often dozens, of tests into the one test,
> interdependencies aren't clear. (The lack of isolation also harms the
> reliability/correctness of the tests, but my point here is that it harms
> readability too.)
> - Certain operations are simply more awkward in a doctest. Reusing common > set
> up and tear down is harder (you can't just subclass or import from a doctest),
> so people tend to just keep adding to the existing doctest file rather than
> making a new file when they should. Again the result is long, rambling
> doctest files that are both poor tests and poor documentation.
> - It's harder to name and talk about a specific failing test case when your test
> case is example 27, about, oh, half-way down foo-feature.txt, compared to
> "FooTestCase.test_feature_with_all_knobs_turned_off". This is frustrating
> when communicating with fellow developers about a test failure, and is
> exacerbated by the lack of correlation between reported line numbers in errors
> and actual line numbers in the file. (This also mildly irritating when trying
> to run just the failing test for purposes of debugging and fixing the
> failure.)
I would be very happy if you read about this proposal, try the module,
perform your own assessments, and provide us (the project) with some
feedback about whether this module can be a solution (alternative) to
all or part of the issues mentioned above and also those mentioned in
[12], [14], [15], [16], and maybe somewhere else (but I think this
message is longer than I wanted... up to this point). The module also
helps to separate the match made for individual doctest examples in
the context of the unittest run.
If we all like it -initially I do... ;-) - and think it is helpful, it
could be incorporated into stdlib as an alternative to the actual
unittest API offerred by doctest since Python 2.4. But perhaps there
is still a long way before this, in case it ever happens.
It would be nice to have some (positive / negative) feedback from
you... it feels too bad being so lonely in this world ;-/ ... and I
think this list is the right place.
---------- Forwarded message ----------
From: Olemis Lang <olemis at gmail.com>
Date: 12 août 2008 09:08
Subject: An OO API for doctest / unittest integration...
To: python-dev at python.org
Hello... this is my first post to this list and...
I would like to introduce myself by proposing a module for inclusion
(hopefully in a "near future") among the standard libraries (modules)
offered by Python (of course the idea is to present it here, promote
the debate and if "we all like it", well... distribute it together
with Python). I will be as concise as possible all the way through
this message so as to avoid writing a long speech (and therefore a
lengthy message). Further details can be found in an article published
"a few months ago" [1].
The module (its name... dutest) has the same goals considered to
develop the unittest API provided by doctest since Python 2.4.
However it includes other important features which make the
difference. Some of them are the following:
* A novel object-oriented API programmers can use in order to
retrieve test cases out of interactive sessions in the form
doctest examples. In other words, novel test loaders find and
put test cases together, just like unittest test loaders do.
Nowadays the unittest API included in doctest offers the functions
DocFileSuite and DocTestSuite for this purpose, therefore lacking
in object orientation.
* The match made for different doctest examples during a doctest run
is stored separately by instances of TestResult. Nowadays
the unittest API included in doctest stores the whole doctest
report at once into test results. This has some "drawbacks".
Firstly, it is difficult to know what is wrong during the test
(since to kinds of reports are intermingled). Besides several
summaries can be found throughout test reports. When building test
analysis tools, this also means that further parsing should be
done so as to discover the descriptions of individual failures ...
(further and more detailed analysis and explanations can be found
in the aforementioned article [1])
* A few more features...
The module is actually employed for testing the pyOOP package
(under development by the FLiOOPS project [2]). You can simply
download dutest [3] and try it out, or you can also check out the
trunk [4] doing something like
$ svn co https://flioops.svn.sourceforge.net/svnroot/flioops/py/trunk .
install the package doing something like
$ setup.py install
$ setup.py clean
Optionally (if you like to see a much more real test report) you
can also download the modules ipdbc [5], PyDBC [6], and
PyContracts [7], and install them.
Next, run pyOOP test suite from the Python interpreter doing something
like
{{{
#!python
try:
from oop.test import check_oopdbc
except ImportWarning:
from oop.test import check_oopdbc
oop.test.test_oopdbc.TESTLOG_FILE = '/path/to/log/file'
check_oopdbc()
}}}
and "see" the output report (e.g. $ vi /path/to/log/file).
Finally, any bug or bizarre behavior you find, please submit it to the
bug tracking system of the FLiOOPS project [8]. We will fix it "as
soon as we can". If you have any suggestion or need an additional
feature, please submit a feature request to the FLiOOPs project [9]...
and/or share it with us here in this list... ;) as well as any
comments about the whole thing.
I hope you like it :)
----
[1] O. Lang, "Doctest and unittest… now they'll live happily forever"
(2008) The Python Papers, vol. 3, no. 1, pp 38:51, ISSN 1834-3147.
[2] FLiOOPS project home page
(https://flioops.sourceforge.net)
[3] Download page of module dutest
(https://sourceforge.net/project/showfiles.php?group_id=220287&abmode=1)
[4] pyOOP SVN trunk
(https://flioops.svn.sourceforge.net/svnroot/flioops/py/trunk)
[5] Yet another invariant/pre-/postcondition design-by-contract
support module,
Dmitry Dvoinikov
(http://www.targeted.org/python/recipes/ipdbc.py)
[6] PyDBC -- Design by Contract for Python 2.2+,
Daniel Arbuckle
(http://www.nongnu.org/pydbc/)
[7] Contracts for Python (PEP 316),
Terence Way
(http://www.wayforward.net/pycontract/)
[8] Bug tracking system of the FLiOOPS project
(https://sourceforge.net/tracker/admin/?atid=1049023&group_id=220287)
[9] Feature Request tracker system of the FLiOOPS project
(https://sourceforge.net/tracker/admin/?atid=1049026&group_id=220287)
[10] Send patches to the FLiOOPS project
(https://sourceforge.net/tracker/admin/?atid=1049025&group_id=220287)
---------------- End of forwarded message ----------------
[11] [TIP] Doctest or unitest?
(http://lists.idyll.org/pipermail/testing-in-python/2007-March/000077.html)
[12] [TIP] Doctest or unitest?
(http://lists.idyll.org/pipermail/testing-in-python/2007-March/000123.html)
[13] [Python-Dev] An OO API for doctest / unittest integration...
(http://mail.python.org/pipermail/python-dev/2008-August/081761.html)
[14] [TIP] Doctest or unitest?
(http://lists.idyll.org/pipermail/testing-in-python/2007-March/000104.html)
[15] [TIP] Best way to do setup/teardown on a per-test basis using doctest?
(http://lists.idyll.org/pipermail/testing-in-python/2007-June/000323.html)
[16] [TIP] Best way to do setup/teardown on a per-test basis using doctest?
http://lists.idyll.org/pipermail/testing-in-python/2007-July/000326.html
--
Regards,
Olemis.
More information about the testing-in-python
mailing list