[TIP] Result protocol / pass-fail-error

Jesse Noller jnoller at gmail.com
Mon Apr 13 10:06:12 PDT 2009


On Mon, Apr 13, 2009 at 12:34 PM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
> Jesse Noller wrote:
>>>>
>>>> 2> There is the generic concept of a "test result" - in my mind, this
>>>> is the simple "PASS/FAIL/ERROR" concept of a test which is completed.
>>>>
>>>>
>>>
>>> Agreed - there should be a single result element with values PASS / FAIL
>>> /
>>> ERROR.
>>>
>>
>> Building on this, and Scott's point -
>>
>> test_cases:
>>    test:
>>        id: STR
>>        result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED (or
>> TIMEOUT)
>>        result_id: STR: 0 | 1 | 255 | ... others
>>
>
> You do not need to define all the possible values for result.  Instead, you
> can define the most common, but allow other user-defined values.
>

Agreed

> The value of result is mostly just a way to group things together for
> reporting.  In pandokia, we used pass, fail, error, disabled (i.e.
> explicitly turned off), and missing (i.e. no report).  In the latest schema,
> we allow installation-defined values.
> If you need a test result of "happy" which is better than fail but not as
> good as pass, you can do that.  The reporting system does not know what
> "happy" means -- it just knows that it can show you all the tests that had
> "happy" as a result.  If you add "sad" later (worse than happy, but not as
> bad as fail?), it's just another column on the report, another value to
> query on.

For *my* case, I need to be able to communicate what a result "means"
to the system above the consumer of the result file and take action
(not just report). I see no reason that consumers could not define
custom codes.

> Similarly, if you don't distinguish between "fail" and "error", you don't
> have to report any records with status=error and you don't have to have that
> column on your report.

True.

> So, the real question is:  Which of these status values is so important as
> to be defined in the protocol, and which can be considered site-specific?

See my followup, I outlined something like this:

 result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED

KILLED might be site-specific, although I could argue that it can be
generally defined as "executed by the test runner for any reason
(timeout, etc)"



More information about the testing-in-python mailing list