[TIP] Nosejobs!

[Jesse Noller]

On Thu, Apr 9, 2009 at 2:27 PM, Doug Philips <dgou at mac.com> wrote:
> IIRC, at PyCon 2009, the py.test folks demo'd that very thing. IIRC2, you didn't need to have py.test already installed on the remote system, but I'm not sure.
>
> Given the cross-polination between nose and py.test I would be suprised if this ability weren't already on the radar of the nose developers. Ideally it could be abstracted out for other testing frameworks to use too.
>
> -Doug
>
>

I was there :)

What py.test is doing is significantly different then what I am trying
to do. In my case, it has to be fully asynchronous, distributed
amongst multiple physical sites, and so on. Additionally, there's a
part I left out which runs above this - something I called "Facelift".

"Facelift" would be a combination continuous integration build server
/ test dispatcher. The various facelift servers would detect changes
in a given source tree. Once a change is detected a build+unit tests
would run (I'd probably use pony-build for this). Once a build is
promoted from "goop" -  it would then be sent to $clients for
deployment (install) - these are the nosejob clients.

Facelift would identify possible nosejob servers, and if none are
available, would provision some from a pool (possibly, say, mosso or
EC2 servers) and bring them online. Facelift would then push the job.

A "job" would ultimately be something like (build, test zip, config) -
nosejob would handle the execution/management of the job - it would
push the UUID for the job back to facelift.

Once a job is complete, a result package is ready: facelift can be
notified of the now-ready-results and harvest the results from the
client. The results are parsed (into a database for visualization) and
the build is then tagged with the appropriate tag, which triggers
promotion, and possibly new test runs.

A build is essentially "managed" by Facelift - it evolves from goop
into deployable - each job is a suite of sorts, let's say:

build
unit tests
smoke tests
regression tests
performance tests
scalability tests
resilience tests

Each of these is a tag - a build has a tag placed on it which means at
the end (when a build has all of the tags) it's passed through all
levels of testing. A build is "published" (ready for internal people)
when it's passed the first 3, ready for "ship" when it's passed all of
them.

This is all pie-in-the-sky right now; I'm starting simple with
something *not* reliant on SSH (which is a fickle mistress), the
initial nosejob server I am working on uses SimpleXMLRPCServer to
manage communications. It enqueues jobs as they come in, so you can
pile up many requests on a given client.

This is glossing over a lot of my other "wants" - adding plugins for
nose to support non-python tests, common results formatting, and so
on. My goals for an initial job-taker (nosejobs) are simple for now.

And if py.execnet/py.test/any other open source work can save me time:
I plan on absconding with it/using it. I'm not a fan of wheel
recreation. Yes, I have looked at alternatives - my goal is to make
this as automatic as possible, support multiple languages for tests,
and *not* require that a test be written *for* the framework. A test
should be as simple as a shell script, or as complex as you want, it
should not have to know about the framework running it.

-jesse