[TIP] Result protocol / data content

Jesse Noller jnoller at gmail.com
Mon Apr 13 19:14:25 PDT 2009


On Mon, Apr 13, 2009 at 9:16 PM, Doug Philips <dgou at mac.com> wrote:

> It seems we have two levels of data here:
> -- Data having to do with an individual test.
> -- Data having to do with the aggregate of some number of individual
> tests.

no, we have one thing representing a container which contains enough
information to identify, debug and otherwise provide a fantastic
amount of information about a group of tests executed.

> Either we use the wire protocol to handle the nesting/aggregation of
> individual tests, or make it explicit data in the aggregate's data.

What? I'm using JSON/YAML syntax to our advantage.

> What, in my projection of the Tutisian-sense/defintion-of-simple, is
> the simplest thing we need:
> Test identification and test result.
>    Foo: PASS
>    Bar: FAIL
>

Cool. Then just in your result file, which is simple a subset of the
"entire file" define:

test_cases:
    test:
        id:
        result:

And leave the rest NULL/None. Done.

> --everything-- else should be optional, not included if not needed.

Individual implementations can exclude or include information (and
extend for more) as needed. I don't functionally see what the problem
is in defining a hierarchy of data.

> On thing I like about Pandokia's tactic is the concept of "if this
> specific test doesn't indicate a value for 'x', use this one". OTOH,
> that could just as well be provided, per test, by the test-running-
> infrastructure,

Use None, to represent an empty value; If a test omits it, but the
consumer needs it, the executor can pad it.

> so I'm not sure it is needed. Most of the additional
> data I'm OK with, but I don't see the need for all this nesting
> brouhaha, seems over engineered. I can see one level of nesting for
> the attributes associated with each test. Additional nesting within
> should be implementation-extension. Really, if I want stdout, stderr,
> trace-foo, whatever, do I really need a separate container for it?

Error: Can not parse. I don't see how *minor* nesting, in order to
compartmentalize data which is alike, or will be repeated is a *bad*
thing. I simply do not grok what is wrong with:

run data: { test_cases: {} }

The test cases are a result of the run, ergo, they are children of a
given run. Data about the test is a child of the test. This seems like
a natural fit, and pretty clear.

> Consumers (after the wire-protocol-reader is done) should ignore
> anything they don't understand, but pass it through if they are also
> generators.

And logically, they would: extract what you need but pass the rest on.

Let me point out I don't see tests themselves emitting this
information - I see the thing *running* the tests emitting this data,
however, tests could emit data - but it would be a tiny subset of the
overall, only relating to the container for this test:

test:
    id: my.silly.TestCase
    result: PASS
    additional:
        anything the test wants to add here

The executor would then do something like:

run['test_cases'].append(result)

Or something akin to that. Most of the metadata is managed (rightly
so) by the thing *running* the tests. The important thing is that it's
valid and can be inserted into the collection for the run.

If you wanted, sure - the test could emit the full record: but that
would require passing in a lot of context to the test itself, so it
could emit all of the relevant data + the run metadata.

jesse



More information about the testing-in-python mailing list