[TIP] test results tracking, etc...

[Robert Collins]

On Sun, 2009-04-05 at 10:03 -0400, Douglas Philips wrote:
> On or about 2009 Apr 5, at 3:08 AM, Robert Collins indited:
> > You may find subunit (http://launchpad.net/subunit) useful then. It
> > doesn't support inconclusive, but there are a few ways you could get  
> > it
> > to do so, for instance by using the 'raise a special exception and
> > having your result object handle the specially' pattern.
> 
> Ok, I'm a bit confused. subunit seems to be all about using more  
> processes to run the tests which seems like overkill to just get a few  
> new status codes.
> On bit I didn't mention in my background notes, we link with some C++  
> glue code to talk to the custom drivers needed to talk to our  
> hardware. For various technical reasons it would be painful to have  
> subprocesses (even just one) because of initial startup discovery. I  
> don't like it, but it is not a part of our testing environment over  
> which we can exert control.

Subunit could be described as a xUnit structured TAP, its why I
mentioned it - your other reply to Michael "Understood. From the
discussions at pycon there seemed to be interest  
in having a common results format across many testing tools so that  
"we" could build up a suite/eco-system of result processing tools that  
would be decoupled from the particular tool(s) being used to generate  
the results." confirms what I thought.

The subprocess aspect is overemphasised in the docs, I'm filing a bug
right after this mail about that.

> > I think this should be done purely in python, with the use of  
> > subunit or
> > something like it to get external results into python, where we can  
> > use
> > nice things like any of the existing gui runners to report on results.
> 
> Well, our tests themselves and all of our framework are written in  
> Python, so I'm not sure what you're suggesting here.

TAP is a very perl centric protocol in my experience. From your
mentioning TAP I assumed you would be grabbing one of the mature TAP
frameworks that do test reporting, which is what prompted my comment :).
Further to that, I mean that the python xUnit model for describing tests
is nice to work with and keeping that model even when doing later
analysis means that you can do immediate-run-and-analyse and also
run,store, and analyse simply by changing what 'suite' you plug into
your runner.

A little code may help make this clear - here is 'run an report' in
typical unittest:

suite = make_suite()
result = TextTestResult(sys.stdout)
suite.run(result)

here is 'run and log to a file' with subunit
suite = make_suite()
stream = file('tests.log', 'wb')
result = TestProtocolClient(stream)
suite.run(result)
stream.close()

here is 'process a log and report on it' with subunit
stream = file('tests.log', 'rb')
suite = ProtocolTestCase(stream)
result = TextTestResult(sys.stdout)
suite.run(result)
stream.close()

HTH,
Rob

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090406/11c3dd51/attachment.pgp