[TIP] Result protocol / pass-fail-error
Jesse Noller
jnoller at gmail.com
Mon Apr 13 10:06:12 PDT 2009
On Mon, Apr 13, 2009 at 12:34 PM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
> Jesse Noller wrote:
>>>>
>>>> 2> There is the generic concept of a "test result" - in my mind, this
>>>> is the simple "PASS/FAIL/ERROR" concept of a test which is completed.
>>>>
>>>>
>>>
>>> Agreed - there should be a single result element with values PASS / FAIL
>>> /
>>> ERROR.
>>>
>>
>> Building on this, and Scott's point -
>>
>> test_cases:
>> test:
>> id: STR
>> result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED (or
>> TIMEOUT)
>> result_id: STR: 0 | 1 | 255 | ... others
>>
>
> You do not need to define all the possible values for result. Instead, you
> can define the most common, but allow other user-defined values.
>
Agreed
> The value of result is mostly just a way to group things together for
> reporting. In pandokia, we used pass, fail, error, disabled (i.e.
> explicitly turned off), and missing (i.e. no report). In the latest schema,
> we allow installation-defined values.
> If you need a test result of "happy" which is better than fail but not as
> good as pass, you can do that. The reporting system does not know what
> "happy" means -- it just knows that it can show you all the tests that had
> "happy" as a result. If you add "sad" later (worse than happy, but not as
> bad as fail?), it's just another column on the report, another value to
> query on.
For *my* case, I need to be able to communicate what a result "means"
to the system above the consumer of the result file and take action
(not just report). I see no reason that consumers could not define
custom codes.
> Similarly, if you don't distinguish between "fail" and "error", you don't
> have to report any records with status=error and you don't have to have that
> column on your report.
True.
> So, the real question is: Which of these status values is so important as
> to be defined in the protocol, and which can be considered site-specific?
See my followup, I outlined something like this:
result: STR: PASS | FAIL | ERROR | SKIP | UNKNOWN | KILLED
KILLED might be site-specific, although I could argue that it can be
generally defined as "executed by the test runner for any reason
(timeout, etc)"
More information about the testing-in-python
mailing list