[TIP] Implementation details of x-outcomes WAS: unitteset outcomes (pass/fail/xfail/error), humans and automation

[Robert Collins]

On Fri, 2009-12-18 at 17:47 -0500, Mark Sienkiewicz wrote:


> I don't think it is possible to tunnel custom outcomes through subunit 
> as it exists today.

It's definitely not possible today. I plan to issue an extension to
subunit to permit such tunnelling once we've done the design work to
figure out what such tunnelling needs to be able to represent.

>  The protocol is a line oriented name:value where 
> each possible outcome has its own name. Looking at the protocol spec, it 
> appears that a minimal subunit file looks something like:
> 
> fail: name3
> success: name2
> fail: name1
> banana: tasty

Actually,
test: name3
fail: name3
test: name2
success: nam2
test: name1
fail: nam1
banana: tasty

the test: lines introduce a test, this provides scope for tagging,
allows duration of tests to be measures, and provides for detection of
tests that cause the testing process to crash or hang.

> This reports results for three tests (name1, name2, name3), but the line 
> that says "banana" is not a test status for the test named "tasty". 
> According to the protocol spec in the README, it is just "unexpected 
> lines which are not interpreted by the parser". It looks like a new test 
> status would need change to the protocol and a patch to each program 
> that reads the protocol.

Adlibbing, I'd lean towards:
outcome: tasty banana

or perhaps status, as you suggest immediately after ;).

> Of course, since subunit is still at version 0.0.4, maybe the protocol 
> could be changed before it gets too entrenched. If you allow lines like:
> 
> status: word test_label DETAILS
...
> Of course, a processing system has to know what to do when it sees 
> another status that it has never seen before, but that is implied by the 
> idea that statuses might be user-defined.

precisely! Thanks for looking at how this might work within subunit.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20091219/ae9db582/attachment.pgp>