[TIP] Implementation details of x-outcomes WAS: unitteset outcomes (pass/fail/xfail/error), humans and automation
robertc at robertcollins.net
Fri Dec 18 15:01:43 PST 2009
On Fri, 2009-12-18 at 17:47 -0500, Mark Sienkiewicz wrote:
> I don't think it is possible to tunnel custom outcomes through subunit
> as it exists today.
It's definitely not possible today. I plan to issue an extension to
subunit to permit such tunnelling once we've done the design work to
figure out what such tunnelling needs to be able to represent.
> The protocol is a line oriented name:value where
> each possible outcome has its own name. Looking at the protocol spec, it
> appears that a minimal subunit file looks something like:
> fail: name3
> success: name2
> fail: name1
> banana: tasty
the test: lines introduce a test, this provides scope for tagging,
allows duration of tests to be measures, and provides for detection of
tests that cause the testing process to crash or hang.
> This reports results for three tests (name1, name2, name3), but the line
> that says "banana" is not a test status for the test named "tasty".
> According to the protocol spec in the README, it is just "unexpected
> lines which are not interpreted by the parser". It looks like a new test
> status would need change to the protocol and a patch to each program
> that reads the protocol.
Adlibbing, I'd lean towards:
outcome: tasty banana
or perhaps status, as you suggest immediately after ;).
> Of course, since subunit is still at version 0.0.4, maybe the protocol
> could be changed before it gets too entrenched. If you allow lines like:
> status: word test_label DETAILS
> Of course, a processing system has to know what to do when it sees
> another status that it has never seen before, but that is implied by the
> idea that statuses might be user-defined.
precisely! Thanks for looking at how this might work within subunit.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 197 bytes
Desc: This is a digitally signed message part
More information about the testing-in-python