[TIP] Result protocol (was: Nosejobs!)

[Jesse Noller]

On Sat, Apr 11, 2009 at 10:55 AM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
>> 1> There is the "object" a given *unit test* framework (subunit, nose,
>> py.test, unittest, etc) tracks as a result. These are execution
>> frameworks, and so the result object they track is an artifact of of
>> writing tests into the framework.
>
> How does this relate to the protocol? Do you mean that the test framework
> used should be an element in the protocol? Probably not important for the
> protocol itself.
>

It could be, but I brought it up as one of the many points floating
around in the various threads - I think someone brought up the
TestResult object that comes from a TestCase, personally, I wouldn't
force the framework used should be a default part of the protocol
namespace.

>> 2> There is the generic concept of a "test result" - in my mind, this
>> is the simple "PASS/FAIL/ERROR" concept of a test which is completed.
>>
>
> Agreed - there should be a single result element with values PASS / FAIL /
> ERROR.

Building on this, and Scott's point -

test_cases:
    test:
        id: STR
        result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED (or TIMEOUT)
        result_id: STR: 0 | 1 | 255 | ... others

Ideally, the ID is a simple int, ala Unix error codes. In the past,
I've used something like ERROR: 300, SKIP: 301, or other. I try not to
use ones low enough to be relevant to real exit codes as it could be
useful to actually pass back the exit code of the test command itself
- on the fence about that though.

>
>> 3> Then there is the concept of a "result object" which can be passed
>> over the network to "something" such as a results collector. There has
>> been discussion around using something like a flat-text file
>> pushed/pulled using key:value pairs, JSON/YAML objects, or something
>> else.
>>
>
> This is the protocol itself. If we are to hammer out a protocol we should
> avoid talking about objects. It conflates the protocol with the system used
> to deal with it.
>

Yup, sorry about that. Did not conflate intentionally, only lack-of-coffee-ally

> For a test protocol representing results of test runs I would want the
> following fields:
>
> Build guid
> Machine identifier
> Test identifier: in the form "package.module.Class.test_method" (but a
> unique string anyway)
> Time of test start
> Time taken for test (useful for identifying slow running tests, slow downs
> or anomalies)
> Result: PASS / FAIL / ERROR
> Traceback
>

I think traceback is an artifact of the type of test, in the case of
non-python tests, you might not have one. You would however
(hopefully) have stderr. So -1 on traceback and +1 on error_out or
something akin to that.

build_id: STR
machine_id: STR (uuid?)
test_cases:
    test:
        id: STR
        result: STR
        result_id: STR
        start: FLOAT (time.time())
        stop: FLOAT (time.time())
        total_time: FLOAT (seconds)

> Anything else? What about collecting standard out even if a test passes?
> Coverage information?

Hmm, stdout and stderr are a PITA - I dealt with a test in the past
that created log files several mb in size, logjamming it in this would
suck. How about something like "last N lines" for both? Or we make it
optional, ala something like:

build_id: STR
machine_id: STR (uuid?)
test_cases:
    test:
        id: STR
        result: STR
        result_id: STR
        start: FLOAT (time.time())
        stop: FLOAT (time.time())
        total_time: FLOAT (seconds)
        additional:
            coverage_info: Big str
            stdout: Big Str
            stderr: Big str

We let end-users add on additional tags/information in the addition
section for each test case, and the consumer of the protocol can
ignore, look for, or do something else with that information?
Additional must be defined, but needs not contain anything.

> We sometimes have to kill wedged test processes and need to push an error
> result back. This can be hard to associate with an individual test, in which
> case we leave the test identifier blank.

Don't you know what test (suite) was being run when you killed it?
That was you can do a best-effort associated with it. Blank
identifiers should be fine, but could give consumers heartburn unless
we offer a best practice.

> Extra information (charts?) can be generated from this data. If there is a
> need to store additional information associated with an individual test then
> an additional 'information' field could be used to provide it.

Yup, let me extend this with some other wants:

job_id: UUID
recv_time: FLOAT
build_id: STR
machine_id: STR (uuid?)
execution_line: STR

run_stats:
    total_tests: INT
    total_pass: INT
    total_fail: INT
    total_other: INT
    start_time: FLOAT
    stop_time: FLOAT
    total_time: FLOAT

test_cases:
    test:
        id: STR
        result: STR
        result_id: STR
        start: FLOAT (time.time())
        stop: FLOAT (time.time())
        total_time: FLOAT (seconds)
        additional:
            coverage_info: Big str
            stdout: Big Str
            stderr: Big str

extended:
    requestor_id: INT
    artifacts_uri: <your path here>
    artifacts_chksum: MD5
    configuration_file: (the config file passed in in my case)


Again, I'm straw manning, so feel free to knock it down.

jesse