[TIP] unitteset outcomes (pass/fail/xfail/error), humans and automation

Robert Collins robertc at robertcollins.net
Tue Dec 15 12:29:56 PST 2009


On Tue, 2009-12-15 at 09:45 -0500, Olemis Lang wrote:

> And also to treat then uniformly . I mean I am using testing framework
> X and suddenly switch to TF Y and that's transparent for all apps
> processing the test results (CI slaves or whatever ;o) ...

I don't think that such a change can be transparent in all cases. The
more transparent the better, but as a trivial example, does the
framework use test 'ids' or some human description, or the object path
to the test?

> > Well, on the standardising output front, have you looked at subunit ? :)
> > It aims at precisely that, with current included support for outputtin
> > it from pyunit, shunit, cppunit, 'check' (A C xUnit implementation),
> > converting to junit's xml format (which nearly all CI tools and many
> > IDE's understand) :)
> >
> 
> ... the serialization of the test results is another subject , but could be
> simpler if all TFs report test results following a well-known
> (model | pattern | standard | specification ...)

CouldaWouldaShoulda :- testing frameworks don't all follow xUnit, and
even if they did, xUnit is a moving target: for all that people have
written books, xUnit is evolving and changing continually in the many
implementations of the basic idea.

> > I think the API interactions for the things you mention are well
> > addressable, though in a few different ways.
> 
> Yes, but let's go step by step and focus on test outcomes first , isn't it ?
> ;o)

well, second yes, but front of the queue now.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20091216/dfee2d14/attachment.pgp>


More information about the testing-in-python mailing list