[TIP] TAP vs subunit

[Robert Collins]

On 16 May 2016 at 21:45, Remko Tronçon <remko at el-tramo.be> wrote:
> Hi Robert,
>
>> TAP is/was a very good fit for Perl, but its really awkward for most
>> everything else - and AFAICT it hasn't really changed to fit the more
>> object orientated approach possible in other languages.
>
> Could you elaborate why OO doesn't fit in TAP?

In Python every test has a unique id - testobject.id() - this lets you
select a single test to re-run in the event of failure. As far as I
know TAP does not model this: you have a test script + an integer
offset within the output of that script, and no way to say 'run just
this one failing case'.

>> Consider e.g.
>> https://github.com/remko/pycotap/blob/master/pycotap/__init__.py#L132
>> which doesn't provide semantic information (that the error is an
>> unexpected success) to TAP consumers.
>
> Isn't this a problem with 'unittest' not providing any more information,
> rather than a problem with TAP or pycotap? Also, which semantic
> information would you expect? There's nothing to report in this case afaict?

'unexpected success' is semantic information, provided by Python and
emitted as plain text (not semantic information) by the TAP serializer
because TAP only understands 'ok' or 'not ok' for a test outcome.

>> I'd use junitxml (https://pypi.python.org/pypi/junitxml) which is both
>
> One downside (apart from XML not being as trivial to generate as TAP) is
> that it isn't streamable, so you can't get live updates. Might not be
> necessary for you.

For things running in CI, you can use a Y adapter in your test stack
to provide pretty-printed console output  during execution, as well as
a junit file at the end.

For instance, using subunit as the realtime protocol:

python -m subunit.run discover . | tee test-log.subunit | subunit2pyunit
subunit2junitxml < test-log.subunit > test-output.xml

This will stream and report approximately the full range of outcomes
Python unittest supports.

> TAP certainly isn't perfect, but it is a pragmatic solution that does the trick
> in my case.

Sure,:). You might enjoy subunit, which I started - gosh over a decade
ago now - because TAP seemed like such a poor fit for Python (and
because doing full process isolation of Python tests via TAP really
didn't make sense).

-Rob