[TIP] Result protocol

[Scott David Daniels]

Jesse Noller wrote:
> ...
> test_cases:
>     test:
>         id: STR
>         result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED (or TIMEOUT)
>         result_id: STR: 0 | 1 | 255 | ... others
>
> Ideally, the ID is a simple int, ala Unix error codes. In the past,
> I've used something like ERROR: 300, SKIP: 301, or other. I try not to
> use ones low enough to be relevant to real exit codes as it could be
> useful to actually pass back the exit code of the test command itself
> - on the fence about that though.
>   
I don't know what UNKNOWN, KILLED, and TIMEOUT mean (in some "precise" 
way).  I'd
like to see that defined for a protocol.
> build_id: STR
> machine_id: STR (uuid?)
> test_cases:
>     test:
>         id: STR
>         result: STR
>         result_id: STR
>         start: FLOAT (time.time())
>         stop: FLOAT (time.time())
>         total_time: FLOAT (seconds)
>
>   
The presence of start, stop, and total_time seems to ask for
normalization.  I'd drop one (since it can be calculated from the others).
Perhaps drop stop, since total may have more precision.


--Scott David Daniels
Scott.Daniels at Acm.Org