[TIP] test results tracking, etc...

[Robert Collins]

Sorry for you getting this twice Doug, I forgot my work address isn't
subscribed to tip.

On Sun, 2009-04-05 at 21:38 -0400, Douglas Philips wrote:
> On or about 2009 Apr 5, at 5:01 PM, Robert Collins indited:
> > Subunit could be described as a xUnit structured TAP, its why I
> > mentioned it ...
> >
> > The subprocess aspect is overemphasised in the docs, I'm filing a bug
> > right after this mail about that.
> 
> Thanks, does that mean it can be used without subprocesses getting  
> involved?

Yes.

>  I admit that I pretty much stopped reading at the point  
> because I knew that having to use subprocesses would be a show stopper  
> for me.

Sure. So AIUI you have N pieces of hardware, running tests. You want to
collect the tests to a central place as well as showing them as they run
in the buildbot UI.

The way you'd use TAP is to have the tests output TAP, and then use a
TAP reporter in the buildbot UI and also in the central place. The
machines running the tests do not need to spawn new processes at all.

Subunit works in the same way; the support for subprocesses in it is for
a particular use case - one you can completely ignore.

> > TAP is a very perl centric protocol in my experience. From your
> > mentioning TAP I assumed you would be grabbing one of the mature TAP
> > frameworks that do test reporting, which is what prompted my  
> > comment :).
> 
> Ah, I only mentioned it because the name was being slung around at  
> pycon as something to look into. Better to reuse than reinvent. ;)

:).

> > A little code may help make this clear - here is 'run an report' in
> > typical unittest:
> >
> > suite = make_suite()
> > result = TextTestResult(sys.stdout)
> > suite.run(result)
> >
> > here is 'run and log to a file' with subunit
> > suite = make_suite()
> > stream = file('tests.log', 'wb')
> > result = TestProtocolClient(stream)
> > suite.run(result)
> > stream.close()
> >
> > here is 'process a log and report on it' with subunit
> > stream = file('tests.log', 'rb')
> > suite = ProtocolTestCase(stream)
> > result = TextTestResult(sys.stdout)
> > suite.run(result)
> > stream.close()
> 
> I'm a little bit confused about the differences between the 2nd and  
> third examples, perhaps because I don't get ProtocolTestCase taking a  
> file parameter and TextTestResult also taking a file. Some part of the  
> plumbing here is confusing me.
> 
> --Doug

The second example takes a stream to write raw test data to. Its useful
for getting test activity from one machine to another (the stream could
be a network socket). For instance:
import unittest
import subunit

class TestFoo(unittest.TestCase):
  def test_foo(self):
    self.fail("argh")
suite = TestFoo("test_foo")
stream = file('tests.log', 'wb')
result = subunit.TestProtocolClient(stream)
suite.run(result)
stream.close()

If you run this, then 
$cat tests.log
test: __main__.TestFoo.test_foo
failure: __main__.TestFoo.test_foo [
Traceback (most recent call last):
  File "foo.py", line 6, in test_foo
    self.fail("argh")
AssertionError: argh
]

You can see that what has been output is 'raw' information about the
test.

And the third example is used to take this raw data and convert it back
to python objects. The reason TextTestResult takes a file is simply for
showing you what it does - its actually unittest._TextTestResult. The
result object in the third example could equally be something to convert
the test activity to database records or $whatever.

Specifically ProtocolTestCase is the deserialiser for subunit, and
TestProtocolClient is the serialiser.

-Rob
-- 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.idyll.org/pipermail/testing-in-python/attachments/20090406/682c15b5/attachment.pgp