[TIP] Result protocol

[Michael Foord]

Robert Collins wrote:
> I'm sorry to rabbit on about subunit but this conversation really does
> seem to be focusing on reinventing the wheel.
>   

Do you have any links to documentation. I can't see links on the 
launchpad site and quick google search turned up nothing. (Well I found 
this - http://www.robertcollins.net/unittest/subunit/README )

What format is your text result protocol? It looks non-standard, 
requiring a custom parser for any tool that wants to interoperate with 
it (something we are keen to avoid).

Michael


> On Sat, 2009-04-11 at 10:10 -0400, Jesse Noller wrote:
>   
>> On Sat, Apr 11, 2009 at 12:39 AM, Douglas Philips <dgou at mac.com> wrote:
>>
>>     
>>> Personally my interest is in something much simpler. it isn't the test
>>> that I want to stream back, it is the test running infrastructure that
>>> will keep track of what is going on and reporting back on things. But
>>> even that can get wedged/confused (esp. if it is a simple thing
>>> without much smarts), so I don't want the wire level protocol to be
>>> any more fragile than it has to be. As when I say streaming, I don't
>>> necessarily mean keeping a connection open, it could be done RESTfully
>>> for that matter.
>>>       
>> I thought I'd pull out this discussion around a "results protocol"
>> into it's own thread since it's interesting in and of itself. I'm just
>> throwing stuff against the wall here.
>>     
>
>   
>> There's plenty of discussion to be had on this subject - let me see if
>> I can outline some of the discussion points:
>>
>> 1> There is the "object" a given *unit test* framework (subunit, nose,
>> py.test, unittest, etc) tracks as a result. These are execution
>> frameworks, and so the result object they track is an artifact of of
>> writing tests into the framework.
>>     
>
> A couple of points I think are important:
>   * subunit is *not* a unit test framework. It is a 'Result protocol'.
>     It is what you are designing. I assume you haven't looked at it at
>     all or you wouldn't have classified it as a test framework. It may
>     not be the best, or other folk may build it differently - thats
>     fine, at a certain point this is bikeshedding. Anyhow, its
>     multilanguage (C/C++/python/shell at this time of writing), simple
>     (human readable).
>     https://launchpad.net/subunit 
>
>   * nose/unittest/py.test (AFAIK - Holger correct me) do *not* have a
>     object representing the result of a single test. Instead the result
>     of a single test is the interaction between the test and the result
>     object (which could be better described as a reporter).
>
>   
>> 2> There is the generic concept of a "test result" - in my mind, this
>> is the simple "PASS/FAIL/ERROR" concept of a test which is completed.
>>     
>
> There are lots of tests standards around; xfail, unexpected pass and
> skip are needed. (xfail is a test that isn't expected to work yet,
> unexpected pass is when that test actually works, and skip is for tests
> you do not want to run based on some condition). Some of these can be
> done in the reporting framework, but if you do that it needs more
> metadata and becomes more tied to a situation preventing aggregation and
> sharing. 
>
>   
>> 3> Then there is the concept of a "result object" which can be passed
>> over the network to "something" such as a results collector. There has
>> been discussion around using something like a flat-text file
>> pushed/pulled using key:value pairs, JSON/YAML objects, or something
>> else.
>>     
>
> We don't have such an object today; pandokia's wire protocol, subunit,
> and I imagine py.test has one for its parallelising reporting will all
> be creating it in some manner.
>
>   
>> Personally, I feel as if 1 and 2 are largely things which depend on
>> the exact executor, and can easily be parsed/translated into point 3
>> objects.
>>     
>
> To me it is the other way around: its easy to write a wire protocol to
> match [most] object protocols, but its very hard to write one when the
> object protocol doesn't support what you want to accomplish.
>
> Now Michael says:
>
>   
>> For a test protocol representing results of test runs I would want
>> the following fields:
>>
>> Build guid
>> Machine identifier
>> Test identifier: in the form "package.module.Class.test_method" (but
>> a unique string anyway)
>> Time of test start
>> Time taken for test (useful for identifying slow running tests, slow 
>> downs or anomalies)
>> Result: PASS / FAIL / ERROR
>> Traceback
>>     
>
> subunit does all this today; it is limited by the TestCase->TestResult
> protocol :(.
>
> Here is an example, Build guid and machine identifier I would just tag,
> the time: instruction acks as a clocking signal which allows duration to
> be inferred. (this is hand typed, so the traceback is..odd :). It shows
> a 37 minute duration for this test to execute.
>
> tag: build-2009-04-12-12:23:00, machine-test-alpha1
> test: Library.Engine...DependencyTest.test_method
> time: 2009-04-12 12:23:00
> time: 2009-04-12 13:00:00
> fail: Library.Engine...DependencyTest.test_method [
>   AssertionError:
>     1 != 0 in demo.py 
> ]
>
>
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog