[TIP] RFC: additional data in unittest TestResult outcomes

[Michael Foord]

This is really interesting Robert, but there is a lot of detail. Can you 
provide an abstract / summary?

It seems to be a protocol for object serialisation on test results 
composed of several parts.

Methods on TestResult for attaching information.
Standard ways of storing that information.
A text serialisation protocol for communicating result information.

Is that correct?

Michael

Robert Collins wrote:
> So, this is one of a number of things I need to raise for discussion; I
> think the aggregate set should become a PEP for unittest improvements
> and be taken to python-dev. However, I want to be sure that it works
> first :)
>
> I've been meaning to write a single compelling argument for these
> things, but I think the perfect has become the enemy of the good; so I'm
> going to simply discuss them piecemeal as time & reminders permit.
>
> In unittest, when a test has an outcome (e.g. it failed) it calls
> result.testOutcome(test, ...)
>
> depending on the outcome, the ... might be empty, a textual reason, or a
> backtrace triple.
>
> This is easy to use and all, but its rather limiting. A common thing I
> see unittest extensions do is to overload startTest and stopTest to
> capture stdout and stderr. Another way that that is sometimes done is
> inspect the test object for magic attributes which have log files or
> other additional data.
>
> I think we should make the protocol for doing this be part of the core.
>
> The simplest thing I've come up with so far is to simply allow an
> arbitrary number of data for a test. So, a TestCase might say:
> result.testFailedExtended(self, {'traceback': self.exc_info(), 'log':
> self.log_contents(), 'stdout': self.captured_stdout.getvalue()})
>
> I think this would be very easy for results to deal with.
> _TextTestResult would simply stringify each item when running in verbose
> mode. A GUI test runner could display the keys of the dict in a tree
> view. Elements it knows about it could look for and handle specially.
>
> More importantly not every bit of data a test can gather [and provide
> for debugging failures] makes sense as a string. For instance,
> callgraphs are argurable best presented as kcachegrind files, for
> loading into kcachegrind. Likewise a DB or B+Tree from a failed run is
> binary and useful to examine on disk and with external tools, not as
> something to show in the console.
>
> If we want to be able to ship test results over the network (as py.test,
> Pandokia, Subunit and others either do, or enable the doing-of), then we
> should in our object protocol support this. So I think it would be great
> to add a little bit more metadata to allow these use cases without
> utterly swamping someone watching an interactive test run. 
>
> There is some prior art here - Pandokia uses a flat namespace of
> strings. bzr's test runner uses objects (it does the 'ask the test case
> for extended data' trick). Drizzle leaves debug directories on disk. I'm
> not sure what py.test does.
>
> In the interests of getting it wrong I suggest using a content object
> rather than a plain string; giving that object a content-type for now,
> and seeing how it goes/if people like it. If its too limiting we can add
> a content-encoding; if its too broad we can talk about just-strings.
>
> What would be _really cool_ is if the py.test, nose & Pandokia folk can
> comment on whether this:
>  - fits with what they do [even if its spelt differently today]
>  - is too broad or too limiting.
>
> Implementation-wise, as far as I can see neither wsgi nor
> SimpleHTTPServer actually define a Content object type, so we can't just
> reuse. The email Mime support is all about messages; perhaps it would be
> appropriate to use, but it doesn't really look like it to me.
>
> The goal in the code below is to allow:
>  - in python processing to be clear and introspect tracebacks as
>    some TestResults may do today. If we think no TestResults actually
>    do that today, then we can simply say that all information about a 
>    test outcome has to be stringlike (but still attach content types to
>    permit sensible display without guessing)
>  - allow other in python processing to be equally nice
>  - allow extended processing and integration with external services
>    like Pandokia to be easy to achieve.
>
> class ContentType(object):
>     """A content type from http://www.iana.org/assignments/media-types/
>
>     :ivar type: The primary type, e.g. "text" or "application"
>     :ivar subtype: The subtype, e.g. "plain" or "octet-stream"
>     :ivar parameters: A dict of additional parameters specific to the
>         content type.
>     """
>
>     def __init__(self, primary_type, subtype, parameters=None):
>          ...
>
> class Content(object):
>     """Content that has been attached to a test outcome.
>
>     :ivar content_type: The content type of this Content.
>     """
>
>     def iter_bytes(self):
>         """Return an iterator over bytestrings for this content."""
>
>
> class Traceback(Content):
>     """Python backtraces.
>
>     :ivar exc_info: The sys.exc_info for the exception.
>     :ivar _failureException: The Exception the generating test used
>         for raising Assertions.
>     """
>
>     def __init__(self, exc_info, failureException):
>         self.content_type = ContentType("text", "x-python-traceback")
>         self.exc_info = exc_info
>         self._failureException = failureException
>
>     def _exc_info_to_string(self):
>         exctype, value, tb = self.exc_info
>         # Skip test runner traceback levels
>         while tb and self._is_relevant_tb_level(tb):
>             tb = tb.tb_next
>         if exctype is self._failureException:
>             # Skip assert*() traceback levels
>             length = self._count_relevant_tb_levels(tb)
>             return ''.join(traceback.format_exception(exctype, value,
>                tb, length))
>         return ''.join(traceback.format_exception(exctype, value, tb))
>
>     def iter_bytes(self):
>         return [self._exc_info_to_string()]
>
>
> compatibility
> =============
>
> TestCase objects calling Extended outcomes should handle the outcome
> method being absent and fallback to calling the existing matching
> outcome. The original outcome should not be called if the Extended one
> existed.
>
> So - thoughts?
>
> In case its not obvious, I'm volunteering to see how this works in
> practice in at least a few places. Concretely, one of the issues
> reported by Ronny Pfannschmidt in getting Subunit <-> Nose integration
> was in being able to do the stdout/stderr capture (and Subunit's
> outcomes are an exact mapping of unittests where the outcome exists at
> all, something I want to keep)... so this, to me, is an ideal test case.
>
> -Rob
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>   


-- 
http://www.ironpythoninaction.com/