[TIP] Result protocol / pass-fail-error
dgou at mac.com
Mon Apr 13 13:25:44 PDT 2009
>>> result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED
>>> KILLED might be site-specific, although I could argue that it can be
>>> generally defined as "executed by the test runner for any reason
>>> (timeout, etc)"
>SKIPPED and DISABLED are relatively close to one another. In my case,
>if a resource is missing, the tests may self-SKIP - they could also be
>forcibly skipped by the user
DISABLED means an intentionality on the part of a human.
SKIP means an intentionality on the part of the test code itself.
>> I also have MISSING, which means we did not receive a result for this test.
That seems to be a derived value, based on higher level knowledge of a "expected to get a status for..." kind of thing.
Probably not, I agree, something reported directly from a test run, but as the stream of results is processed and transformed, definitely something that could be added by an agent.
>That makes sense, in terms of JSON/YAML/etc - I would fill in all
>fields, period - even if they were with None values. This makes
>parsing it easier.
UGH, I thought we were talking about that in another thread. If not, then IMHO, only the test "id" and result should be required, --everything-- else should be optional/not-present. I'll continue my reply in the other thread later tonight.
>> I'm a little clear on UNKNOWN. It means we 1) we did run the test, 2) we do
>> not know the status that resulted, 3) but not because there was an error.
>> Does your system do this, or did you just include it for completeness?
>The latter - I would probably never use UNKNOWN, but I can see it as a
>last-ditch "wtf" versus using a None/null value.
If KILLED means that the testing infrastructure took action, the UNKNOWN can mean the testing infrastructure noticed an otherwise indescribable phenomina. The difference being a bit subtle, but could cover the case where a test crashes rather than is killed.
ERROR to mean indicates a bug in the test, not the system/device-under-test.
I'd like to see a --simple-- way to extend these status codes. Maybe: Anything not completely UPPERCASE is implementation defined.
More information about the testing-in-python