[TIP] Result protocol / data content
Jesse Noller
jnoller at gmail.com
Mon Apr 13 19:14:25 PDT 2009
On Mon, Apr 13, 2009 at 9:16 PM, Doug Philips <dgou at mac.com> wrote:
> It seems we have two levels of data here:
> -- Data having to do with an individual test.
> -- Data having to do with the aggregate of some number of individual
> tests.
no, we have one thing representing a container which contains enough
information to identify, debug and otherwise provide a fantastic
amount of information about a group of tests executed.
> Either we use the wire protocol to handle the nesting/aggregation of
> individual tests, or make it explicit data in the aggregate's data.
What? I'm using JSON/YAML syntax to our advantage.
> What, in my projection of the Tutisian-sense/defintion-of-simple, is
> the simplest thing we need:
> Test identification and test result.
> Foo: PASS
> Bar: FAIL
>
Cool. Then just in your result file, which is simple a subset of the
"entire file" define:
test_cases:
test:
id:
result:
And leave the rest NULL/None. Done.
> --everything-- else should be optional, not included if not needed.
Individual implementations can exclude or include information (and
extend for more) as needed. I don't functionally see what the problem
is in defining a hierarchy of data.
> On thing I like about Pandokia's tactic is the concept of "if this
> specific test doesn't indicate a value for 'x', use this one". OTOH,
> that could just as well be provided, per test, by the test-running-
> infrastructure,
Use None, to represent an empty value; If a test omits it, but the
consumer needs it, the executor can pad it.
> so I'm not sure it is needed. Most of the additional
> data I'm OK with, but I don't see the need for all this nesting
> brouhaha, seems over engineered. I can see one level of nesting for
> the attributes associated with each test. Additional nesting within
> should be implementation-extension. Really, if I want stdout, stderr,
> trace-foo, whatever, do I really need a separate container for it?
Error: Can not parse. I don't see how *minor* nesting, in order to
compartmentalize data which is alike, or will be repeated is a *bad*
thing. I simply do not grok what is wrong with:
run data: { test_cases: {} }
The test cases are a result of the run, ergo, they are children of a
given run. Data about the test is a child of the test. This seems like
a natural fit, and pretty clear.
> Consumers (after the wire-protocol-reader is done) should ignore
> anything they don't understand, but pass it through if they are also
> generators.
And logically, they would: extract what you need but pass the rest on.
Let me point out I don't see tests themselves emitting this
information - I see the thing *running* the tests emitting this data,
however, tests could emit data - but it would be a tiny subset of the
overall, only relating to the container for this test:
test:
id: my.silly.TestCase
result: PASS
additional:
anything the test wants to add here
The executor would then do something like:
run['test_cases'].append(result)
Or something akin to that. Most of the metadata is managed (rightly
so) by the thing *running* the tests. The important thing is that it's
valid and can be inserted into the collection for the run.
If you wanted, sure - the test could emit the full record: but that
would require passing in a lot of context to the test itself, so it
could emit all of the relevant data + the run metadata.
jesse
More information about the testing-in-python
mailing list