[TIP] Result protocol (was: Nosejobs!)

[Robert Collins]

I'm sorry to rabbit on about subunit but this conversation really does
seem to be focusing on reinventing the wheel.

On Sat, 2009-04-11 at 10:10 -0400, Jesse Noller wrote:
> On Sat, Apr 11, 2009 at 12:39 AM, Douglas Philips <dgou at mac.com> wrote:
> 
> > Personally my interest is in something much simpler. it isn't the test
> > that I want to stream back, it is the test running infrastructure that
> > will keep track of what is going on and reporting back on things. But
> > even that can get wedged/confused (esp. if it is a simple thing
> > without much smarts), so I don't want the wire level protocol to be
> > any more fragile than it has to be. As when I say streaming, I don't
> > necessarily mean keeping a connection open, it could be done RESTfully
> > for that matter.
> 
> I thought I'd pull out this discussion around a "results protocol"
> into it's own thread since it's interesting in and of itself. I'm just
> throwing stuff against the wall here.

> There's plenty of discussion to be had on this subject - let me see if
> I can outline some of the discussion points:
> 
> 1> There is the "object" a given *unit test* framework (subunit, nose,
> py.test, unittest, etc) tracks as a result. These are execution
> frameworks, and so the result object they track is an artifact of of
> writing tests into the framework.

A couple of points I think are important:
  * subunit is *not* a unit test framework. It is a 'Result protocol'.
    It is what you are designing. I assume you haven't looked at it at
    all or you wouldn't have classified it as a test framework. It may
    not be the best, or other folk may build it differently - thats
    fine, at a certain point this is bikeshedding. Anyhow, its
    multilanguage (C/C++/python/shell at this time of writing), simple
    (human readable).
    https://launchpad.net/subunit 

  * nose/unittest/py.test (AFAIK - Holger correct me) do *not* have a
    object representing the result of a single test. Instead the result
    of a single test is the interaction between the test and the result
    object (which could be better described as a reporter).

> 2> There is the generic concept of a "test result" - in my mind, this
> is the simple "PASS/FAIL/ERROR" concept of a test which is completed.

There are lots of tests standards around; xfail, unexpected pass and
skip are needed. (xfail is a test that isn't expected to work yet,
unexpected pass is when that test actually works, and skip is for tests
you do not want to run based on some condition). Some of these can be
done in the reporting framework, but if you do that it needs more
metadata and becomes more tied to a situation preventing aggregation and
sharing. 

> 3> Then there is the concept of a "result object" which can be passed
> over the network to "something" such as a results collector. There has
> been discussion around using something like a flat-text file
> pushed/pulled using key:value pairs, JSON/YAML objects, or something
> else.

We don't have such an object today; pandokia's wire protocol, subunit,
and I imagine py.test has one for its parallelising reporting will all
be creating it in some manner.

> Personally, I feel as if 1 and 2 are largely things which depend on
> the exact executor, and can easily be parsed/translated into point 3
> objects.

To me it is the other way around: its easy to write a wire protocol to
match [most] object protocols, but its very hard to write one when the
object protocol doesn't support what you want to accomplish.

Now Michael says:

> For a test protocol representing results of test runs I would want
> the following fields:
>
> Build guid
> Machine identifier
> Test identifier: in the form "package.module.Class.test_method" (but
> a unique string anyway)
> Time of test start
> Time taken for test (useful for identifying slow running tests, slow 
> downs or anomalies)
> Result: PASS / FAIL / ERROR
> Traceback

subunit does all this today; it is limited by the TestCase->TestResult
protocol :(.

Here is an example, Build guid and machine identifier I would just tag,
the time: instruction acks as a clocking signal which allows duration to
be inferred. (this is hand typed, so the traceback is..odd :). It shows
a 37 minute duration for this test to execute.

tag: build-2009-04-12-12:23:00, machine-test-alpha1
test: Library.Engine...DependencyTest.test_method
time: 2009-04-12 12:23:00
time: 2009-04-12 13:00:00
fail: Library.Engine...DependencyTest.test_method [
  AssertionError:
    1 != 0 in demo.py 
]


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090412/562f3cf9/attachment.pgp