[TIP] Result protocol / pass-fail-error

[Victoria G. Laidler]

Olemis Lang wrote:
>
>> The idea I have is to be consistent with the current test status
>> (ERROR, FAILURE, SUCCESS | PASS | NOT EXPLICITLY FAILED- ) in order to
>> be backward compatible. However allow to specialize the status codes
>> in an OO manner. Firstly by establishing a hierarchy like the
>> following :
>>
>> - ERROR
>> - FAIL
>>  * KILLED
>>  * TIMEOUT
>>  * UNKNOWN
>> - PASS
>>  * SKIP
>>  * WARNING
>>  * DISABLED
>>  * MISSING
>>  * INCONCLUSIVE
>>     
I disagree with the grouping of this hierarchy, and I'm not sure a 
hierarchy buys you enough to be worth the inflexibility (EG, you clearly 
think that "timeout" is a subcase of Fail; I think it's a subcase of 
Error. With a defined hierarchy we have to come to agreement; without 
one, we can each apply the grouping or hierarchy logic ourselves, 
downstream.
>> Secondly by allowing to record failure objects (instead of strings
>> like is done nowadays) inside TestResult.failures, TestResult.errors,
>> and add TestResult.successful (or whatever name you prefer ;)
>>     
Objects are not language-agnostic, which this result protocol is 
intended to be. Test statuses in this protocol cannot be objects.

Changes to TestResult (additions, extensions, patches, whatever) aren't 
really relevant to the discussion.

Vicki