[TIP] Result protocol / pass-fail-error

[Doug Philips]

>>>  result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED

>>> KILLED might be site-specific, although I could argue that it can be
>>> generally defined as "executed by the test runner for any reason
>>> (timeout, etc)"

>SKIPPED and DISABLED are relatively close to one another. In my case,
>if a resource is missing, the tests may self-SKIP - they could also be
>forcibly skipped by the user

My understanding:
    DISABLED means an intentionality on the part of a human.
    SKIP means an intentionality on the part of the test code itself.

>> I also have MISSING, which means we did not receive a result for this test.

That seems to be a derived value, based on higher level knowledge of a "expected to get a status for..." kind of thing.
Probably not, I agree, something reported directly from a test run, but as the stream of results is processed and transformed, definitely something that could be added by an agent.

>That makes sense, in terms of JSON/YAML/etc - I would fill in all
>fields, period - even if they were with None values. This makes
>parsing it easier.

UGH, I thought we were talking about that in another thread. If not, then IMHO, only the test "id" and result should be required, --everything-- else should be optional/not-present. I'll continue my reply in the other thread later tonight.

>> I'm a little clear on UNKNOWN.  It means we 1) we did run the test, 2) we do
>> not know the status that resulted, 3) but not because there was an error.
>>  Does your system do this, or did you just include it for completeness?
>
>The latter - I would probably never use UNKNOWN, but I can see it as a
>last-ditch "wtf" versus using a None/null value.

If KILLED means that the testing infrastructure took action, the UNKNOWN can mean the testing infrastructure noticed an otherwise indescribable phenomina. The difference being a bit subtle, but could cover the case where a test crashes rather than is killed.

ERROR to mean indicates a bug in the test, not the system/device-under-test.

I'd like to see a --simple-- way to extend these status codes. Maybe: Anything not completely UPPERCASE is implementation defined.

-Doug