[TIP] Test reporting in unittest (and plugins)
michael at voidspace.org.uk
Mon Aug 2 07:51:52 PDT 2010
I've been thinking about test reporting and the unittest plugin system.
By default, i.e. if you use the unit2 test runner script or call
unittest(2).main(...), test reporting is done by a combination of the
TextTestRunner and a TextTestResult object.
Reporting of test names / descriptions is done by
TextTestResult.getDescription(test) which uses the string representation
of the test, along with possibly the docstring, to describe the test.
This is all hard to modify from a plugin. If you just want to output
*extra* information then you can use the new messaging API , but if
you want to modify the way test descriptions are output or the way
certain types of errors are reported then you are out of luck.
What you *can* do is take over reporting altogether and silence the
default stream. This doesn't play well with other plugins though, and
will silence the message api (for example). Another alternative would be
to supply a custom TestResult that controls reporting, but again only
one plugin can do this. This is obviously inadequate.
The problem is that the way the testresult is called by
TestCase.run(...) doesn't allow for the description to be modified or
the way that tests are reported to be modified. I don't think that the
desired goal can be achieved without modifying the test result - but
remaining backwards compatible with old test result objects that don't
provide the new api. If anyone has an alternative suggestion I will
listen however. :-)
In previous discussions with Jason Pellerin I had resisted the idea of
changing the TestResult for this purpose, as replacing the TestResult
with a custom one is commonly done. Currently I see no straightforward
My proposal is that we add a new method to the TestResult addDetail (or
similar) that stores a data structure representing the outcome. The
default data structure will be available in the stopTest event for
plugins to modify. The textual parts of the result, like the
description, will have a list for extra bits to be prepended / appended
so that plugins can add extra information without stomping on
information added by other plugins.
A test result with an addDetail method will store this data structure
which will be used to display the results at the end and during the test
run when in verbose mode. A test result without this method will be
restricted to the information available through the current API.
A side effect of this will be the allowing of custom outcomes, which are
collected separately and displayed using a different 'letter' in test
runs. Custom outcomes will have to have a "fallback" of one of the
existing outcomes (pass, skip, fail, error) for tools that don't work
with custom outcomes (like junit-xml) or for old result objects.
How does this sound to everyone?
All the best,
..  The messaging API now has the ability to use the strings "quiet",
"normal" and "verbose" as verbosity levels instead of just numbers
thanks to a suggestion from exarkun. I have also added as a feature
request his suggestion to specify arbitrary channels for logging, but it
is hard for me to see how that would fit in with the way that unittest
verbosity currently works. If anyone else would like this please chime
in and we can discuss how it should look. Certainly allowing the user to
specify channels they are interested in and only outputting messages on
enabled channels would be easy. It seems to me that most information
would be lost that way though - even in verbose mode, unless the user
knows which channels are *used* by your specific plugin. I'll see if I
can prototype that in a separate plugin however. If it is popular it can
move into core unittest.
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
More information about the testing-in-python