[TIP] Implementation details of x-outcomes WAS: unitteset outcomes (pass/fail/xfail/error), humans and automation

[Mark Sienkiewicz]

>>> I want to:
>>>  - be able to automate with them: if I add an outcome 'MissingFeature',
>>> I want to be able to add workflow and reports on that in my CI system or
>>> test repository.
>>>
>>>       
>> while being language agnostic ? is that still a requirement ?
>>     
>
> I don't think language agnostic can really apply while we're talking
> about the python unittest API. I will want to figure out how to tunnel
> whatever design we come up with through subunit, in both directions -
> e.g. how to represent a custom outcome in junit in python - but thats my
> problem ;).
>   

I don't think it is possible to tunnel custom outcomes through subunit 
as it exists today. The protocol is a line oriented name:value where 
each possible outcome has its own name. Looking at the protocol spec, it 
appears that a minimal subunit file looks something like:

fail: name3
success: name2
fail: name1
banana: tasty

This reports results for three tests (name1, name2, name3), but the line 
that says "banana" is not a test status for the test named "tasty". 
According to the protocol spec in the README, it is just "unexpected 
lines which are not interpreted by the parser". It looks like a new test 
status would need change to the protocol and a patch to each program 
that reads the protocol.

Of course, since subunit is still at version 0.0.4, maybe the protocol 
could be changed before it gets too entrenched. If you allow lines like:

status: word test_label DETAILS

where word is the name of a user-defined status or one of the existing 
statuses, it would be easy to pass custom statuses through the system. 
For example:

status: fail name3
status: success name2
status: fail name1
status: banana tasty
spinach: yukky

Here, I've added a test named "tasty" that yielded the result "banana", 
but the line about spinach is still unknown text to be ignored.

Of course, a processing system has to know what to do when it sees 
another status that it has never seen before, but that is implied by the 
idea that statuses might be user-defined.

Mark S.