[TIP] Nosejobs!

[Jesse Noller]

On Sat, Apr 11, 2009 at 12:39 AM, Douglas Philips <dgou at mac.com> wrote:
> never said that it needed to know anything about the world around
> itself, though I find the prospect of "watching the surrounding
> directory" rather odd. Is that where these charts and graphs and what
> not come from?

Yes, for example, performance runs in certain type of products. Sure,
you can make a test which does the required things, but solid
assertions are difficult given inherent variability. Ergo, the test
outputs a chart which is part of the overall payload.

> Personally my interest is in something much simpler. it isn't the test
> that I want to stream back, it is the test running infrastructure that
> will keep track of what is going on and reporting back on things. But
> even that can get wedged/confused (esp. if it is a simple thing
> without much smarts), so I don't want the wire level protocol to be
> any more fragile than it has to be. As when I say streaming, I don't
> necessarily mean keeping a connection open, it could be done RESTfully
> for that matter.

Ok; so I don't follow you here. I'm not defining a wire protocol, I'm
defining a basic (let's pick YAML for now) YAML object which
represents a single, or set of test results.

In those results, you have the normal, expected information - name,
runtime, results, and so on - you can nest where required, or not. You
define a few bits for tracebacks/errors or "open text" fields, and
you're done.

You leave that file open enough, so that a test can notify the
executor above it that "hey, add xxx information to your results for
me". That additional information should avoid being overly complex.

That's the first part of the payload; the second is the binary blob,
which contains logs, any additional output, etc. The "regular payload"
and the binary payload are both important to keep.

Also, nothing I've mentioned excludes the concept of using REST-style
APIs/calls to solve this, but I don't get where you're going with it
in this case.

> Never meant to mislead you that it was the tests that had to know
> anything about the surrounding framework.
> As to results formatting, I'm not sure how "human readable" jibes with
> charts and graphs and arbitrary blobs of data in json/yaml/protocol-
> buffers.

Misunderstanding, I don't intend to log-jam binary data into the basic
result, my point earlier was to say a result is a combination of two
things - the "result object" and the "other stuff".

JSON/YAML is very readable, but like with anything, if you do go down
the route of jamming too much cruft into it, it can become just as
dense. I suspect you're looking for the definition of the YAML/JSON
object?

>
> It doesn't seem that your vapour has condensed enough yet, since your
> position seems to still be shifting/settling. We'll see.
>

Nope, it hasn't. I'm rather mercenary with design, I'll gladly set
something I think I think is a good idea aside if something better
comes along.

As it is, and I want to stress this - my initial nosejobs email still
stands, that I am working on now to act as a test bed for some of the
lower level things (such as JSON/blog send back to the caller maybe).
Facelift is very much a longer-term architecture and goal.

> Personally, I'd just like to stay focused on getting the test
> reporting wire level protocol down. That should be an orthogonal
> concern, esp. orthogonal to such issues as 'continuously open socket'
> or "transport via REST" or "file on a USB memory stick", but lots of
> other dimensions as well.

Fair enough, but picking a wire protocol is pretty simple in my mind.
Human readable, extensible. Nothing "special". We have a few simple
options:

YAML
JSON
XML
TEXT

But what is the point, to create a generic singular format for all
test* things to report in?