[TIP] Result protocol / pass-fail-error

[Mark Sienkiewicz]

Jesse Noller wrote:
>>> 2> There is the generic concept of a "test result" - in my mind, this
>>> is the simple "PASS/FAIL/ERROR" concept of a test which is completed.
>>>
>>>       
>> Agreed - there should be a single result element with values PASS / FAIL /
>> ERROR.
>>     
>
> Building on this, and Scott's point -
>
> test_cases:
>     test:
>         id: STR
>         result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED (or TIMEOUT)
>         result_id: STR: 0 | 1 | 255 | ... others
>   

You do not need to define all the possible values for result.  Instead, 
you can define the most common, but allow other user-defined values.

The value of result is mostly just a way to group things together for 
reporting.  In pandokia, we used pass, fail, error, disabled (i.e. 
explicitly turned off), and missing (i.e. no report).  In the latest 
schema, we allow installation-defined values. 

If you need a test result of "happy" which is better than fail but not 
as good as pass, you can do that.  The reporting system does not know 
what "happy" means -- it just knows that it can show you all the tests 
that had "happy" as a result.  If you add "sad" later (worse than happy, 
but not as bad as fail?), it's just another column on the report, 
another value to query on.

Similarly, if you don't distinguish between "fail" and "error", you 
don't have to report any records with status=error and you don't have to 
have that column on your report.

So, the real question is:  Which of these status values is so important 
as to be defined in the protocol, and which can be considered site-specific?

In any case, we need to strictly define what each value means.

Mark S.