[TIP] Result protocol (was: Nosejobs!)

[Robert Collins]

On Sat, 2009-04-11 at 22:25 -0400, Jesse Noller wrote:
> On Sat, Apr 11, 2009 at 6:08 PM, Robert Collins
> <robertc at robertcollins.net> wrote:
> > I'm sorry to rabbit on about subunit but this conversation really does
> > seem to be focusing on reinventing the wheel.
> 
> Now that I'm done cooking for tomorrow and got to spend time with
> subunit, I figured I'd take a stab at replying. Mind you, no - I don't
> want to reinvent the wheel, I just want to make one that fits the car
> I drive.
> 
> > A couple of points I think are important:
> >  * subunit is *not* a unit test framework. It is a 'Result protocol'.
> >    It is what you are designing. I assume you haven't looked at it at
> >    all or you wouldn't have classified it as a test framework. It may
> >    not be the best, or other folk may build it differently - thats
> >    fine, at a certain point this is bikeshedding. Anyhow, its
> >    multilanguage (C/C++/python/shell at this time of writing), simple
> >    (human readable).
> >    https://launchpad.net/subunit
> >
> 
> So, I somehow missed the right code, or I was not grokking what I was
> looking at. I'm familiar with TAP and others in it's ilk, but thanks
> to getting the right URL, I did spend some time with it now :)
> 
> Yes, subunit is essentially something which parses specialized
> output/streams to provide a unified facade of test results. It uses a
> custom, simple syntax so that test results can be extracted, organized
> in a simple way.
>
> Sound about right? :)

About right yes. There are things missing [that is uses the xUnit
interface deliberately, as a key design goal], but at the heart of it
you are very close. There is one thing in particular that is important -
subunit serialises *activity* not results.

> I was pondering this looking at your status codes as well. I think
> that the initial volley I sent out can be slightly condensed, but
> added to. Perhaps:
> 
> PASS | FAIL | ERROR | SKIPPED | NOTSUPPORTED | TIMEOUT
> 
> Timeout is an implicit KILLED, but I dropped KILLED as a it could be
> filed until ERROR or TIMEOUT
> 
> The executor may be able to set some of these (for example, SKIPPED on
> tests which have a WIN32 attr)

If you need timeout then you are approaching the problem in a different
way - and that will make it harder IMO. Note that subunit does not need
timeout because of the bracketing approach taken to describing activity.
An interrupted test is detected automatically (if its is killed), and
you can enforce a timeout policy both wherever the test is executing,
and where you are reading test activity from.

> > We don't have such an object today; pandokia's wire protocol, subunit,
> > and I imagine py.test has one for its parallelising reporting will all
> > be creating it in some manner.
> 
> Yup, this is partially what I've been trying to personally (and here) hash out.

Actually no - you have been talking about wire protocols, not about
objects. And, the wire protocols that have been discussed - except for
TAP and subunit - do not appear to fit _at all_ with the unittest object
model. And TAP itself fits rather badly [see tap2subunit for the
gymnastics needed].

> > To me it is the other way around: its easy to write a wire protocol to
> > match [most] object protocols, but its very hard to write one when the
> > object protocol doesn't support what you want to accomplish.
> 
> I can't disagree here, which is why I am proposing using YAML (let's
> ignore JSON for the moment) and/or XML. It's easily extendable, does
> not require custom parsers, and convey a near unlimited amount of
> information. Additionally, they go on the wire pretty nicely.

But proposing yaml/json/xml *is* talking about wire protocols. I repeat:
 The *interesting* problem is the object protocol, not the wire
protocol.

*****
Until unittest can represent a standard way for
accepting/exposing/accessing/whatever all the activity and data that a
test running remotely may generate, designing a wire protocol is
premature.
*****

> Ok, so here's my first issue; a custom syntax. I don't think we need
> or want one. True, you can add tags with subunit, but I don't know
> that in the case Michael and are talking about, that we want to be
> limited by the constraints of TestCase->TestResult protocol.

So, I've said before that I am open to changing the syntax. Writing a
parser is trivial though - just having a dict for each action from the
wire [which is the most any of the mentioned encodings do] doesn't
provide glue to a given languages unittest equivalent. *No matter what
approach is taken* language specific glue needs to be written. I'm not
trying to handwave away the issue of the current approach needing a
parser written - but honestly, the state machine for the parser is a
small fraction of the total code base.

> I'm also worried about rogue stdout - what about malformed output,
> random [ ] characters, and so on. I know you can code defensively
> around this, and any parser runs this risk - but again, it's another
> reason I wanted to avoid yet another syntax/parser.

The current protocol was deliberately written to be transparent to rogue
stdout and stderr. This allows running a debugger or other things
through it. It shouldn't be a matter of defensive code, rather defensive
design.

> I also want to convey more information, in a more structured format
> from the executor to the collector. What subunit is doing is useful -
> and in fact not to far from what you'd need to put together  to deal
> with tests-in-multiple, or "simpler" languages (such as bash).

So subunit can be extended to convey more data - I just don't want to do
that before being able to do it all in-python. Otherwise there _will_ be
a mismatch, and I can assert from experience that we'll get it wrong and
end up having just-another-not-reusable-protocol. Something that subunit
avoids at the moment.

> Also, why did you put all the code in __init__.py, that make baby sad :(

I was younger then ;). OTOH, pynie (parrot's python compiler) have
recently adopted subunit to run their bootstrap tests, and they just
copied the __init__ into their source tree, which is very convenient for
users.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090412/6d1e283e/attachment.pgp