[TIP] TAP vs subunit

[Robert Collins]

On 16 May 2016 at 13:33, Ben Finney <ben+python at benfinney.id.au> wrote:
> Brian Okken <variedthoughts at gmail.com> writes:
>
>> I'm trying to pull together all of the relevant useful extensions to
>> unittest.
>
> The wider world enjoys the Test Anything Protocol
> <URL:http://testanything.org/>.
>
> This library <URL:https://pypi.python.org/pypi/pycotap/> enables TAP
> output for Python's ‘unittest’ runner.

TAP is/was a very good fit for Perl, but its really awkward for most
everything else - and AFAICT it hasn't really changed to fit the more
object orientated approach possible in other languages. Consider e.g.
https://github.com/remko/pycotap/blob/master/pycotap/__init__.py#L132
which doesn't provide semantic information (that the error is an
unexpected success) to TAP consumers.

There have been a bunch of test protocols around, and I'm sure more
will turn up :). If I was picking one for integration with Jenkins,
I'd use junitxml (https://pypi.python.org/pypi/junitxml) which is both
built into Jenkins, and supports a reasonably rich range of
information. It doesn't yet support subtests (patches gratefully
considered), but since the jenkins model doesn't either, the built in
defacto fall back in unittest will handle that gracefully enough.

For big-data, machine processing, distributed clusters - I'd go for a
robust machine target format - either an adhoc schema in
capnproto/protobufs/messagepack/one-of-https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats

For local use, something that lets you debug in-situ and/or in your
IDE of choice is probably best.

-Rob