[TIP] Implementation details of x-outcomes WAS: unitteset outcomes (pass/fail/xfail/error), humans and automation
sienkiew at stsci.edu
Fri Dec 18 14:47:27 PST 2009
>>> I want to:
>>> - be able to automate with them: if I add an outcome 'MissingFeature',
>>> I want to be able to add workflow and reports on that in my CI system or
>>> test repository.
>> while being language agnostic ? is that still a requirement ?
> I don't think language agnostic can really apply while we're talking
> about the python unittest API. I will want to figure out how to tunnel
> whatever design we come up with through subunit, in both directions -
> e.g. how to represent a custom outcome in junit in python - but thats my
> problem ;).
I don't think it is possible to tunnel custom outcomes through subunit
as it exists today. The protocol is a line oriented name:value where
each possible outcome has its own name. Looking at the protocol spec, it
appears that a minimal subunit file looks something like:
This reports results for three tests (name1, name2, name3), but the line
that says "banana" is not a test status for the test named "tasty".
According to the protocol spec in the README, it is just "unexpected
lines which are not interpreted by the parser". It looks like a new test
status would need change to the protocol and a patch to each program
that reads the protocol.
Of course, since subunit is still at version 0.0.4, maybe the protocol
could be changed before it gets too entrenched. If you allow lines like:
status: word test_label DETAILS
where word is the name of a user-defined status or one of the existing
statuses, it would be easy to pass custom statuses through the system.
status: fail name3
status: success name2
status: fail name1
status: banana tasty
Here, I've added a test named "tasty" that yielded the result "banana",
but the line about spinach is still unknown text to be ignored.
Of course, a processing system has to know what to do when it sees
another status that it has never seen before, but that is implied by the
idea that statuses might be user-defined.
More information about the testing-in-python