[TIP] Result protocol / pass-fail-error

[Olemis Lang]

On Tue, Apr 14, 2009 at 8:24 AM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> Olemis Lang wrote:
>>
>> On Mon, Apr 13, 2009 at 3:25 PM, Doug Philips <dgou at mac.com> wrote:
>>
>>>
>>> I'd like to see a --simple-- way to extend these status codes. Maybe:
>>> Anything not completely UPPERCASE is implementation defined.
>>>
>>
>> The idea I have is to be consistent with the current test status
>> (ERROR, FAILURE, SUCCESS | PASS | NOT EXPLICITLY FAILED- ) in order to
>> be backward compatible. However allow to specialize the status codes
>> in an OO manner. Firstly by establishing a hierarchy like the
>> following :
>>
>> - ERROR
>> - FAIL
>>  * KILLED
>>  * TIMEOUT
>>  * UNKNOWN
>> - PASS
>>  * SKIP
>>  * WARNING
>>  * DISABLED
>>  * MISSING
>>  * INCONCLUSIVE
>>
>
> More like:
>
> - ERROR
> * KILLED
> * TIMEOUT
> * UNKNOWN
> * MISSING
> * INCONCLUSIVE
> - FAIL
> - PASS
> * SKIP
> * WARNING
> * DISABLED
>
> I'm not sure if a heirarchy is right anyway. They all seem like separate
> statuses and the reporting tool can group them if it wants.
>

Well I mentionned the reasons why I said so in the previous message.

I'll mention my ideas:

- fail | pass | error is a model widely adopted, and is the one we
have today ;).
- Provided that IMHO error should signal an error in the test code
itself (not a wrong behavior in the SUT) then I put the statuses
inside FAILURE since:
  * KILLED : If KILLED means that the testing infrastructure took
action then it doesnt mean that my code is not well-written but that
the test was not completed successfully
  * TIMEOUT : same as above
  * UNKNOWN : same as above
  * INCONCLUSIVE : according to the original description the test «did
not FAIL» (so its not a failure), the test code is not wrong either
(so its not an error) but did not produce statistically valid PASS
either (so its not exactly a clean PASS). Therefore that's why I
considered it a PASS.

If you consider that putting'em inside ERROR is better, well that's
fine for me ... and if you think its not a valid suggestion it's ok
anyway ;)

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article: