[TIP] Result protocol / data content

[Jesse Noller]

On Tue, Apr 14, 2009 at 12:13 AM,  <laidler at stsci.edu> wrote:
>
> Maybe I missed it in the flood of email, but could you clarify your
> motivation for wanting to define the result protocol as inherently
> describing an aggregate of tests, rather than an individual test?
> What problem are you trying to solve?

In a executor->test relationship, the executor (in my use case) almost
never runs a single test, even if it did, I would want to treat it as
part of a suite. In my brain, the logical break down looks like:

run->tests->test

So tracking test results as an aggregate of a run of tests makes
sense. This means that higher level consumers can easily do things
like, say:

build_results = [ test.result for test in runs[] ]

Or something akin to that.

> I get that a single case can be represented in this protocol as
> the special case for which N=1, but an aggregate seems to me to be
> a peculiar choice of building block.

It's not a special case, consider for the moment that a test result is
a message, and the format I've proposed is a report on received
messages. If only one message is received, only one message is tracked
and reported by the executor

> What are the advantages of defining an aggregate as the basic
> level; rather than defining a single test result as the basic level,
> and letting downstream critters do the aggregating? Or is the idea
> that you know you want to aggregate so why define two protocols
> (one for a single case and one for a bunch) when the former can
> be a special case of the latter?

Correct. I am assuming data visualization, reporting and aggregation
make sense to define. I don't see the need to have two different
things.

>
> If it is an aggregate, are there issues with size or complexity or
> levels of nesting, that would impose a desirable or practical upper
> limit on the size and structure of the aggregate?

No. The only requirement is that a given entry on a top-level be unique.

> Mark and I have gone round in circles a few times on whether the
> testrun should be the parent of a test, or an identifying attribute
> of the test.
> From the point of view of running the tests, and avoiding the repetition
> of a lot of identical data, yes the parent:child relationship seems obvious.
> But from the point of view of comparing the same test *across runs*
> (which we need to do often, dunno about your case), it's actually easier
> to make it an attribute: otherwise you have to parse through the other
> 599 tests in this run, and the other 612 tests in that run, in order to
> get to the two tests you're actually interested in.

I do do comparisons across runs, but a run is the highest level
entity, when I squirt this data into a database, all I do is point my
ORM at it and say "give me all result objects for this test, across
all runs". Given JSON/YAML, you can also access directly by *name*
result['tests']['mytestname'] to get direct access to a result. No
need to hand-hold the rest, or even worry about them. Sure, the parse
which says load('my file') parses the entire document - but given
you're not writing the parser, who cares?

> So I think it really depends what you want to *do* with this thing. You've
> talked about consumers and generators; can you give some specific examples
> of what these things might be?

I think I glossed over some of this when I was discussing Facelift. In
my myopic view - Facelift takes a handful of tests, tells a slave to
run them. In my case, that will more than likely be nose. Nose then
runs the tests I told it to, generates this JSON document for me, and
ships it back to Facelift. Facelift then inserts this document's
fields into a database.

That's the simplest view - on a test level, I put the power into nose
to parse individual results from tests - in my case, it might be a
file dropped by the test (result.json) given I don't believe in
parsing stdout (I view stdout as an artifact of the test, to be
saved).

jesse