[TIP] Result protocol / data content

[Douglas Philips]

On or about 2009 Apr 13, at 10:14 PM, Jesse Noller indited:

> no, we have one thing representing a container which contains enough
> information to identify, debug and otherwise provide a fantastic
> amount of information about a group of tests executed.

That is one way to put it over the wire, true, but I thought we were  
talking abstractly about the information itself here.



> What? I'm using JSON/YAML syntax to our advantage.


Yes, that is putting a wire-protocol mess into the abstraction at a  
layer it doesn't belong. Keeping them separate is simpler.



> And leave the rest NULL/None. Done.


Not sure what you mean by NULL/None, do you mean that the other fields  
aren't present (what I want) or that they are present with Not- 
Applicable kinds of values?



> Individual implementations can exclude or include information (and
> extend for more) as needed. I don't functionally see what the problem
> is in defining a hierarchy of data.

Not sure what is supposedly gained by a hierarchy where stdout (to use  
an example) has to be nested inside another container when it will not  
conflict with anything at the same level as the container itself. I  
could see doing that if there were going to be a list of containers,  
such that promoting the containee tags to a higher level would make  
them ambiguous. Not sure what you're trying to accomplish here...



> Use None, to represent an empty value; If a test omits it, but the
> consumer needs it, the executor can pad it.

Not what I had in mind. If a test executor doesn't report it, the  
consumer can add it. I want to have simple executors be simple. They  
report what they need to report, they don't have to know about fields  
that they don't care about just so that they can pad them. Or maybe I  
misunderstand.



> run data: { test_cases: {} }
>
> The test cases are a result of the run, ergo, they are children of a
> given run. Data about the test is a child of the test. This seems like
> a natural fit, and pretty clear.

The aggregate can be reified directly, which is that you're  
suggesting, making it a container.
The aggregate can be emergent, extracted through a commonality of the  
test data itself (my parent is "X", for example).
I'm not dead-set against this outermost nesting, but I question the  
need to have nesting inside the test description.




> Let me point out I don't see tests themselves emitting this
> information - I see the thing *running* the tests emitting this data,

I agree that the executor is the source of the "test results" which  
will likely be a combination of stuff supplied by the executor itself  
and stuff supplied by the test. The test should not know or care that  
this is happening, nor be affected by it. I think we agree on this (or  
at least the two of us do :) ).

	-Doug