[TIP] RFC: additional data in unittest TestResult outcomes

Robert Collins robertc at robertcollins.net
Sun Sep 20 21:04:24 PDT 2009

So, this is one of a number of things I need to raise for discussion; I
think the aggregate set should become a PEP for unittest improvements
and be taken to python-dev. However, I want to be sure that it works
first :)

I've been meaning to write a single compelling argument for these
things, but I think the perfect has become the enemy of the good; so I'm
going to simply discuss them piecemeal as time & reminders permit.

In unittest, when a test has an outcome (e.g. it failed) it calls
result.testOutcome(test, ...)

depending on the outcome, the ... might be empty, a textual reason, or a
backtrace triple.

This is easy to use and all, but its rather limiting. A common thing I
see unittest extensions do is to overload startTest and stopTest to
capture stdout and stderr. Another way that that is sometimes done is
inspect the test object for magic attributes which have log files or
other additional data.

I think we should make the protocol for doing this be part of the core.

The simplest thing I've come up with so far is to simply allow an
arbitrary number of data for a test. So, a TestCase might say:
result.testFailedExtended(self, {'traceback': self.exc_info(), 'log':
self.log_contents(), 'stdout': self.captured_stdout.getvalue()})

I think this would be very easy for results to deal with.
_TextTestResult would simply stringify each item when running in verbose
mode. A GUI test runner could display the keys of the dict in a tree
view. Elements it knows about it could look for and handle specially.

More importantly not every bit of data a test can gather [and provide
for debugging failures] makes sense as a string. For instance,
callgraphs are argurable best presented as kcachegrind files, for
loading into kcachegrind. Likewise a DB or B+Tree from a failed run is
binary and useful to examine on disk and with external tools, not as
something to show in the console.

If we want to be able to ship test results over the network (as py.test,
Pandokia, Subunit and others either do, or enable the doing-of), then we
should in our object protocol support this. So I think it would be great
to add a little bit more metadata to allow these use cases without
utterly swamping someone watching an interactive test run. 

There is some prior art here - Pandokia uses a flat namespace of
strings. bzr's test runner uses objects (it does the 'ask the test case
for extended data' trick). Drizzle leaves debug directories on disk. I'm
not sure what py.test does.

In the interests of getting it wrong I suggest using a content object
rather than a plain string; giving that object a content-type for now,
and seeing how it goes/if people like it. If its too limiting we can add
a content-encoding; if its too broad we can talk about just-strings.

What would be _really cool_ is if the py.test, nose & Pandokia folk can
comment on whether this:
 - fits with what they do [even if its spelt differently today]
 - is too broad or too limiting.

Implementation-wise, as far as I can see neither wsgi nor
SimpleHTTPServer actually define a Content object type, so we can't just
reuse. The email Mime support is all about messages; perhaps it would be
appropriate to use, but it doesn't really look like it to me.

The goal in the code below is to allow:
 - in python processing to be clear and introspect tracebacks as
   some TestResults may do today. If we think no TestResults actually
   do that today, then we can simply say that all information about a 
   test outcome has to be stringlike (but still attach content types to
   permit sensible display without guessing)
 - allow other in python processing to be equally nice
 - allow extended processing and integration with external services
   like Pandokia to be easy to achieve.

class ContentType(object):
    """A content type from http://www.iana.org/assignments/media-types/

    :ivar type: The primary type, e.g. "text" or "application"
    :ivar subtype: The subtype, e.g. "plain" or "octet-stream"
    :ivar parameters: A dict of additional parameters specific to the
        content type.

    def __init__(self, primary_type, subtype, parameters=None):

class Content(object):
    """Content that has been attached to a test outcome.

    :ivar content_type: The content type of this Content.

    def iter_bytes(self):
        """Return an iterator over bytestrings for this content."""

class Traceback(Content):
    """Python backtraces.

    :ivar exc_info: The sys.exc_info for the exception.
    :ivar _failureException: The Exception the generating test used
        for raising Assertions.

    def __init__(self, exc_info, failureException):
        self.content_type = ContentType("text", "x-python-traceback")
        self.exc_info = exc_info
        self._failureException = failureException

    def _exc_info_to_string(self):
        exctype, value, tb = self.exc_info
        # Skip test runner traceback levels
        while tb and self._is_relevant_tb_level(tb):
            tb = tb.tb_next
        if exctype is self._failureException:
            # Skip assert*() traceback levels
            length = self._count_relevant_tb_levels(tb)
            return ''.join(traceback.format_exception(exctype, value,
               tb, length))
        return ''.join(traceback.format_exception(exctype, value, tb))

    def iter_bytes(self):
        return [self._exc_info_to_string()]


TestCase objects calling Extended outcomes should handle the outcome
method being absent and fallback to calling the existing matching
outcome. The original outcome should not be called if the Extended one

So - thoughts?

In case its not obvious, I'm volunteering to see how this works in
practice in at least a few places. Concretely, one of the issues
reported by Ronny Pfannschmidt in getting Subunit <-> Nose integration
was in being able to do the stdout/stderr capture (and Subunit's
outcomes are an exact mapping of unittests where the outcome exists at
all, something I want to keep)... so this, to me, is an ideal test case.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090921/a9e12f0b/attachment.pgp 

More information about the testing-in-python mailing list