[TIP] Result protocol

[Robert Collins]

On Sat, 2009-04-11 at 22:52 -0400, Jesse Noller wrote:


> That's your call - I obviously like YAML (or JSON - but I don't
> remember if JSON supports multi-line blobs). However, on a personal
> note - I still won't be able to touch subunit, due to licensing.

Could you enlarge on this?

> > As for the format, it is a very simple line based format - because both
> > human readable and standard don't exist :).
> 
> That's why we're here, right? :)
> 
> > That said, *outputting* it is totally trivial, and outputting is the
> > common case: noone (I hope!) wants to write a test reporting framework
> > purely in shell; but many people will have some tests best written in
> > shell.
> 
> And C, and C++, and Perl, and Java. I think if a code base lives long
> enough, it ends up with tests in every language possible. Which
> reminds me, I think we need a OCaml parser ;)

Au contraire - you need a C serialiser (done, with support for check), C
++ serialiser (done, plugin for cppunit), perl (done, by Jelmer, but he
isn't happy with his API yet so its in a branch indefinitely), Java
(haven't done this yet). 

You only need a parser *when you are reporting* in that language. 

> > A much larger problem, and what I think poses much more of an issue than
> > deserialisation is that all the rich inspection of code people are used
> > to doing locally on their machine - clocking, profiling, debugging,
> > logfiles attached with the traceback, stderr output [when they are not
> > already capturing that to a log file] - has absolutely no home in
> > unittest today - nor does it have a home in TAP (because TAP is even
> > less structured than subunit), nor can the unittest frameworks in C/C++
> > that I have seen handle these things.
> 
> This is why I pretty much chucked the idea of TAP early on, and
> keeping within the unit test rules. I can use a unit test runner to
> provide some of the data, but as you can see in the other emails - I
> want to have a much more detailed report. Additionally, the executor
> component would save off arbitrary test output, which would be linked
> to in that report.

My sanity test for this is "if you can't do it without the wire
protocol, you can't do it with the wire protocol". What I mean is that
if you can't (run the same tests locally, in process, the way unittests
run today in say nose, or even stock unittests) and (get your additional
data) then no matter how clever or extensible the wire protocol, you are
no longer working with unittest. Consider clock events - I haven't
mapped this cleanly into the unittest protocol yet, and because of that
its pretty much useless.

> > The thing to spend time on is how to get this extra information
> > reasonably across to the TestResult such that a wire protocol can even
> > conceptually handle it.
> 
> I think I'm trying to solve something else, but which touches on some
> of what you're talking about. I know that the system I want to build
> has some of the attributes of subunit, and so I would use that as a
> bit of a reference.
> 
> In my case, I don't really want to make everything look like a unit
> test test case - I just want something that collects, runs and reports
> on tests of any ilk, so long as those tests obey some basic rules, not
> to far from what you've done, but again, I don't want to stick to icky
> unit testing rules :)

I want to fix unittest to allow it to do this; I don't care about
reporting [in the sense of 'I have a big database and query it'] - I
care that it is possible for someone to write a collector ('TestResult
subclass') that inserts into a database sensibly and gathers the data
and information we want.

If 'someone' can do that without having to customise the object
protocol, then the standard testing facilities will be sufficient for
lots of things like pandokia to be written *and* interoperate - and
because I will keep subunit up to date with the unittest API (which
reminds me I need to make sure the skip support is 2.7/3.1 compatible)

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090412/4499313e/attachment.pgp