[TIP] Result protocol / problems being solved

[Laura Creighton]

In a message of Mon, 13 Apr 2009 22:18:58 EDT, laidler at stsci.edu writes:
<snip>
>2. I want to be able to easily distinguish the following:
>   2A. Broken software
>   2B. Broken tests
>   2C. Broken execution environments
>
>and distinguish them from deliberately (by human or machine) omitted test
>s.
>
>Implications for result protocol: Status should distinguish between Fail 
>(implies broken software) and Error (implies broken test or execution env
>ironment); and possibly further distinguish between "error during test" a
>nd "error attempting to set up test" (might distinguish between broken te
>st and broken environment, although it might be sufficient to use the env
>ironment-related bookkeeping from case 1 for this).

<snip>

>Did I miss anything?
>
>Vicki

Some things I've read last week ....

Some people have tests that are expected to run for a particular amount
of time.  They want a special way to tell when the test didin't run
for that amount of time.

Some people want a way to indicate that they expect a test to fail
in a certain way, so that when it fails in a different way they can
make those differences stand out.

Some people are testing hardware.  I think that Error (meaning
broken test or execution env) won't be enough for them.  They need to
distinguish between 'my hardware didn't work properly' and 'something
is wrong in this test' and 'this hardware didn't have the correct
software loaded on it before we even tried to run this test'.

Whether this last is a special case of the second, I am not sure.

Laura