[TIP] About support for warnings in TestCase [WAS] Determine test status in tearDown method

[Olemis Lang]

On 10/21/09, holger krekel <holger at merlinux.eu> wrote:
> On Wed, Oct 21, 2009 at 08:06 -0500, Olemis Lang wrote:
>  > ...
>
> >
>  > The only thing I can mention so far is that once upon a time I had the
>  > need to keep track of some conditions that were not precisely errors,
>  > but didn't look like total success either (just like a fuzzy measure
>  > of test success, more or less ;o) . I was benchmarking the support
>  > provided by some DBC implementations for Py and sometimes I wanted to
>  > have a result like e.g. «it is consistent with contracts semantics but
>  > doesn't support custom messages to describe the failure». What I did
>  > was to add support for warnings in instances of test cases. Following
>  > that model anybody can keep track of such warnings by issuing a call
>  > just like `tc.warn(msg, ...)` , and you could do it anywhere, even
>  > inside `tearDown` method.
>
>
> I like the idea of systematically allowing test code to signal warnings.

Yes ... you can find the implementation I mention here [1]_. A test
can contain multiple warnings since they accumulate . I had further
ideas like :

  - Integrate it with `warnings` mdl, so that if a code (probably beyond test
    code itself, e.g. deprecation warnings) triggers a system warning then
    the testing framework collects this event inside the current active TC
  - Integrate it with `logging` module ...
  - Integrate it with `dutest` . In fact there's some kind of support for this
    I remember I was able to do something like ;o)

{{{
#! python

>>> __tester__.warn(msg, ...)
}}}

and the enhancements I mentioned above would allow `doctest`-minded
people to do something more stdlib-ish like

{{{
#!python

>>> warnings.warn(msg, ...)
>>> log.warning(msg, ...)
}}}

But, the way I see it, there's a lot of overlapping between `warnings`
module and `logging` module. I don't like the former too much because
it's global and manipulates global data, and this is a real problem
for testing code since may lead fragile tests. OTOH it it possible to
isolate the impact of warnings by using `logging` module and write
more reliable tests (but this is another subject ...)

>  It is then a matter of reporting how/when to present it.
>

I had to extend TestRunner and TestResult in order to do that and ,
since TCs having warnings are reported as successful TCs , I also had
to override `addSuccess` (and probably something inside TestRunner ;o)
in order to replace `ok` label by `WARNINGS` label. You get more or
less verbose output (i.e. warning details) using the usual options in
`unittest` mdl

;o)

.. [1] trunk/utils/testing.py @ FLiOOPS @ SF
        (https://sourceforge.net/apps/trac/flioops/browser/py/trunk/utils/testing.py#L38)

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article: