[TIP] Test reporting in unittest (and plugins)

Michael Foord michael at voidspace.org.uk
Mon Aug 2 07:51:52 PDT 2010

Hello all,

I've been thinking about test reporting and the unittest plugin system. 
By default, i.e. if you use the unit2 test runner script or call 
unittest(2).main(...), test reporting is done by a combination of the 
TextTestRunner and a TextTestResult object.

Reporting of test names / descriptions is done by 
TextTestResult.getDescription(test) which uses the string representation 
of the test, along with possibly the docstring, to describe the test.

This is all hard to modify from a plugin. If you just want to output 
*extra* information then you can use the new messaging API [1], but if 
you want to modify the way test descriptions are output or the way 
certain types of errors are reported then you are out of luck.

What you *can* do is take over reporting altogether and silence the 
default stream. This doesn't play well with other plugins though, and 
will silence the message api (for example). Another alternative would be 
to supply a custom TestResult that controls reporting, but again only 
one plugin can do this. This is obviously inadequate.

The problem is that the way the testresult is called by 
TestCase.run(...) doesn't allow for the description to be modified or 
the way that tests are reported to be modified. I don't think that the 
desired goal can be achieved without modifying the test result - but 
remaining backwards compatible with old test result objects that don't 
provide the new api. If anyone has an alternative suggestion I will 
listen however. :-)

In previous discussions with Jason Pellerin I had resisted the idea of 
changing the TestResult for this purpose, as replacing the TestResult 
with a custom one is commonly done. Currently I see no straightforward 

My proposal is that we add a new method to the TestResult addDetail (or 
similar) that stores a data structure representing the outcome. The 
default data structure will be available in the stopTest event for 
plugins to modify. The textual parts of the result, like the 
description, will have a list for extra bits to be prepended / appended 
so that plugins can add extra information without stomping on 
information added by other plugins.

A test result with an addDetail method will store this data structure 
which will be used to display the results at the end and during the test 
run when in verbose mode. A test result without this method will be 
restricted to the information available through the current API.

A side effect of this will be the allowing of custom outcomes, which are 
collected separately and displayed using a different 'letter' in test 
runs. Custom outcomes will have to have a "fallback" of one of the 
existing outcomes (pass, skip, fail, error) for tools that don't work 
with custom outcomes (like junit-xml) or for old result objects.

How does this sound to everyone?

All the best,

Michael Foord

.. [1] The messaging API now has the ability to use the strings "quiet", 
"normal" and "verbose" as verbosity levels instead of just numbers 
thanks to a suggestion from exarkun. I have also added as a feature 
request his suggestion to specify arbitrary channels for logging, but it 
is hard for me to see how that would fit in with the way that unittest 
verbosity currently works. If anyone else would like this please chime 
in and we can discuss how it should look. Certainly allowing the user to 
specify channels they are interested in and only outputting messages on 
enabled channels would be easy. It seems to me that most information 
would be lost that way though - even in verbose mode, unless the user 
knows which channels are *used* by your specific plugin. I'll see if I 
can prototype that in a separate plugin however. If it is popular it can 
move into core unittest.


READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

More information about the testing-in-python mailing list