[TIP] Result protocol

Jesse Noller jnoller at gmail.com
Sat Apr 11 14:27:34 PDT 2009


On Sat, Apr 11, 2009 at 3:46 PM, Scott David Daniels
<Scott.Daniels at acm.org> wrote:
> Jesse Noller wrote:
>>
>> ...
>> test_cases:
>>    test:
>>        id: STR
>>        result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED (or
>> TIMEOUT)
>>        result_id: STR: 0 | 1 | 255 | ... others
>>
>> Ideally, the ID is a simple int, ala Unix error codes. In the past,
>> I've used something like ERROR: 300, SKIP: 301, or other. I try not to
>> use ones low enough to be relevant to real exit codes as it could be
>> useful to actually pass back the exit code of the test command itself
>> - on the fence about that though.
>>
>
> I don't know what UNKNOWN, KILLED, and TIMEOUT mean (in some "precise" way).
>  I'd
> like to see that defined for a protocol.

UNKNOWN: Unknown/un-identifiable exit, either it checked out (and
wasn't killed) or the status could not be gotten. Could also be None

KILLED: The executor killed the child test, this could be due to
taking too long, sudden preemption, or some other thing.

TIMEOUT: Test timed out - you could possibly use this instead of
KILLED to define a test which has exceeded a built in timer (say, 3
hours) and had to be killed.

>>
>> build_id: STR
>> machine_id: STR (uuid?)
>> test_cases:
>>    test:
>>        id: STR
>>        result: STR
>>        result_id: STR
>>        start: FLOAT (time.time())
>>        stop: FLOAT (time.time())
>>        total_time: FLOAT (seconds)
>>
>>
>
> The presence of start, stop, and total_time seems to ask for
> normalization.  I'd drop one (since it can be calculated from the others).
> Perhaps drop stop, since total may have more precision.
>

Personally, I'd drop total, stop and start would be fine w/me.



More information about the testing-in-python mailing list