[TIP] Result protocol

Jesse Noller jnoller at gmail.com
Sat Apr 11 11:39:01 PDT 2009


On Sat, Apr 11, 2009 at 2:35 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:

>> Building on this, and Scott's point -
>>
>> test_cases:
>>    test:
>>        id: STR
>>        result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED (or
>> TIMEOUT)
>>        result_id: STR: 0 | 1 | 255 | ... others
>>
>> Ideally, the ID is a simple int, ala Unix error codes. In the past,
>> I've used something like ERROR: 300, SKIP: 301, or other. I try not to
>> use ones low enough to be relevant to real exit codes as it could be
>> useful to actually pass back the exit code of the test command itself
>> - on the fence about that though.
>>
>>
>
> Having result *and* result_id seems like a recipe for confusion (what if
> they contradict). Keeping it as a string keeps it human readable.

Fair enough, I can jam the ID (Yay numbers!) into the additional
section if I really want it.

>> [snip...]
>>
>
> But with most testing tools the traceback *is* the diagnostic information.
> It isn't *obvious* where it should go in the schema you have below.
>

Could we call it something other than traceback? Otherwise, I'm OK
with putting it in the test: section, I was on the fence about it.

>> Hmm, stdout and stderr are a PITA - I dealt with a test in the past
>> that created log files several mb in size, logjamming it in this would
>> suck. How about something like "last N lines" for both? Or we make it
>> optional, ala something like:
>>
>> build_id: STR
>> machine_id: STR (uuid?)
>> test_cases:
>>    test:
>>        id: STR
>>        result: STR
>>        result_id: STR
>>        start: FLOAT (time.time())
>>        stop: FLOAT (time.time())
>>        total_time: FLOAT (seconds)
>>        additional:
>>            coverage_info: Big str
>>            stdout: Big Str
>>            stderr: Big str
>>
>>
>
> OK - although I would still rather see one obvious place for the traceback
> (whatever it may be called) for failure information.

See above, I'm iffy on calling it traceback though.



More information about the testing-in-python mailing list