[TIP] Result protocol

Robert Collins robertc at robertcollins.net
Sat Apr 11 15:38:54 PDT 2009


On Sat, 2009-04-11 at 23:14 +0100, Michael Foord wrote:
> 
> 
> Do you have any links to documentation. I can't see links on the 
> launchpad site and quick google search turned up nothing. (Well I
> found 
> this - http://www.robertcollins.net/unittest/subunit/README )

Old :(.

http://bazaar.launchpad.net/%7Esubunit/subunit/trunk/annotate/head%
3A/README

> What format is your text result protocol? It looks non-standard, 
> requiring a custom parser for any tool that wants to interoperate
> with 
> it (something we are keen to avoid).

Standards happen when people agree on a format :). I do appreciate the
point that using (say) json means people don't need to write a low level
parser - but they still need to add semantic interpretation to whatever
low level vehicle was used. 

I'm also happy to change subunit to address concerns [but not without
some debate first, amongst others samba4 is using the current wire
protocol to run their tests and it would be nice to not introduce a
compatibility break]. If we agree that json really is better I could
move the current protocol into a deprecated mode.

As for the format, it is a very simple line based format - because both
human readable and standard don't exist :).

That said, *outputting* it is totally trivial, and outputting is the
common case: noone (I hope!) wants to write a test reporting framework
purely in shell; but many people will have some tests best written in
shell.

And, you only need to write a parser at most once for each language -
once you have the parser emitting events from it you can plugin that
into whatever code you have in the language. I should in fact write a
lex/yacc pair for it, which would make it even easier for people.

A much larger problem, and what I think poses much more of an issue than
deserialisation is that all the rich inspection of code people are used
to doing locally on their machine - clocking, profiling, debugging,
logfiles attached with the traceback, stderr output [when they are not
already capturing that to a log file] - has absolutely no home in
unittest today - nor does it have a home in TAP (because TAP is even
less structured than subunit), nor can the unittest frameworks in C/C++
that I have seen handle these things.

The thing to spend time on is how to get this extra information
reasonably across to the TestResult such that a wire protocol can even
conceptually handle it.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090412/1e3a3e0a/attachment-0001.pgp 


More information about the testing-in-python mailing list