[TIP] test results tracking, etc...

Robert Collins robertc at robertcollins.net
Sun Apr 5 00:08:12 PDT 2009

On Sun, 2009-04-05 at 00:33 -0400, Douglas Philips wrote:
> I don't know if raw TAP will work since we have more than just PASS/ 
> FAIL for our results. We have PASS, FAIL, SKIP, INELIGIBLE and  
> INCONCLUSIVE. Inconclusive means that while the test didn't fail, it  
> also did not run long enough to produce a statistically valid
> result.  
> As I mentioned above, SKIP and INELIGIBLE we're probably going to  
> collapse into one state, so that would leave us with four possible  
> test results.

You may find subunit (http://launchpad.net/subunit) useful then. It
doesn't support inconclusive, but there are a few ways you could get it
to do so, for instance by using the 'raise a special exception and
having your result object handle the specially' pattern.

It has a tap2subunit filter which could be used to import your existing
results, and one of the features its users wanted is 'tags' which would
be a good way to export your keywords to the result log.

> I am hoping that a common test reporting system would still be  
> something we could use and/or contribute to, but regardless we are  
> going to have pursue a solution. I'd personally prefer to not have
> to  
> re-invent the wheel. :)

I think this should be done purely in python, with the use of subunit or
something like it to get external results into python, where we can use
nice things like any of the existing gui runners to report on results.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090405/92ef36e6/attachment.pgp 

More information about the testing-in-python mailing list