[TIP] Nosejobs!

[Jesse Noller]

On Fri, Apr 10, 2009 at 10:58 PM, Douglas Philips <dgou at mac.com> wrote:
>
>> You don't let tests directly write to the object you're going to send back
>> to the system.
>
> Agreed, you want some kind of sanitizer, etc. But that adds complexity...
>

What are you writing the framework for, my dog? You tell me you want
to sanitize results, but you want to handle them as a stream? What's
wrong with run, have them dump what happened, sanitize that data and
send it back? It's not complex. The tests don't even need to be
programmed to work in your environment, *you* just need to know how to
get what you need *from* the tests.

Example: A simple 10 line bash script which tests, say, a login.

In my world, the test doesn't need to know *anything* about the world
around it, only that it should run, and here's some data about the
thing it should run against. It runs, it can log as much as it likes
(I'm watching it's stdout, stderr and anything it adds to the
surrounding directory) and then exit 0 or 1 or XX depending on if it
passed or failed.

That's the simple case: If you want to make it fancier, you don't put
burden on the test writer - you accentuate the piece above that (the
executor) to know more about this class of test if at all possible.
You parse what you want from it's logs, and you add it to the payload
for this tests result in the bevy of results you're sending back.

Yes, this puts more of the burden on the framework writer - you have
to allow for plugins, you have to realize some tests are dumb, and so
on, but it's complexity you only need to do once.

>
>> Doing that means that yes, you can have tests crap on your message
>> back to the mothership - no matter the format, that will always cause
>> pain.
>
> "Test failed - Conversion from EBCDIC has been declined. Recommend
> sterilizing remote system and retrying." :)
>

You don't let your tests talk past the guy controlling them on the
local machine: he communicates up, not the test. He sanitizes the
data, chooses which action to take (kill, destroy, enslave) and then
passes it back. You don't let test send malformed garbage all
willy-nilly over your network.

Really, I just wanted to say willy-nilly.

>
>> Anything which makes something which must be parsed by another
>> language/entity will have some brittle pieces. I like YAML/JSON->Flat
>> text->XML in that order, but again, I'm half tempted to do it with
>> protocol buffers, because it's cool, and shiny.
>>
>> http://code.google.com/apis/protocolbuffers/
>
> Yawn. Yet another IDL/Marshalling spec. "Must be good" because it comes with
> all that Google shi...ne on it.
> seems the opposite of simple, but at this point I'm interested in going in a
> different direction anyways. Will keep an eye on what you're doing though.

No, must be good because it has some nice attributes, like cleanliness
and other things - again, it's an idea, and not my first when it comes
to results formatting (I really like human readable). Ultimately; I do
not want the tests to have to know what my framework "wants" - I want
it to run, and somehow provide me a result. I'll handle result
correlation/collection.

Sorry my rudimentary vaporous design specification doesn't fit your needs. *sad*

-jesse