[TIP] Nosejobs!

[Jesse Noller]

On Thu, Apr 9, 2009 at 2:51 PM, Jesse Noller <jnoller at gmail.com> wrote:
> On Thu, Apr 9, 2009 at 2:27 PM, Doug Philips <dgou at mac.com> wrote:
>> IIRC, at PyCon 2009, the py.test folks demo'd that very thing. IIRC2, you didn't need to have py.test already installed on the remote system, but I'm not sure.
>>
>> Given the cross-polination between nose and py.test I would be suprised if this ability weren't already on the radar of the nose developers. Ideally it could be abstracted out for other testing frameworks to use too.
>>
>> -Doug
>>
>>
>
> I was there :)
>
> What py.test is doing is significantly different then what I am trying
> to do. In my case, it has to be fully asynchronous, distributed
> amongst multiple physical sites, and so on. Additionally, there's a
> part I left out which runs above this - something I called "Facelift".
>
> "Facelift" would be a combination continuous integration build server
> / test dispatcher. The various facelift servers would detect changes
> in a given source tree. Once a change is detected a build+unit tests
> would run (I'd probably use pony-build for this). Once a build is
> promoted from "goop" -  it would then be sent to $clients for
> deployment (install) - these are the nosejob clients.
>
> Facelift would identify possible nosejob servers, and if none are
> available, would provision some from a pool (possibly, say, mosso or
> EC2 servers) and bring them online. Facelift would then push the job.
>
> A "job" would ultimately be something like (build, test zip, config) -
> nosejob would handle the execution/management of the job - it would
> push the UUID for the job back to facelift.
>
> Once a job is complete, a result package is ready: facelift can be
> notified of the now-ready-results and harvest the results from the
> client. The results are parsed (into a database for visualization) and
> the build is then tagged with the appropriate tag, which triggers
> promotion, and possibly new test runs.
>
> A build is essentially "managed" by Facelift - it evolves from goop
> into deployable - each job is a suite of sorts, let's say:
>
> build
> unit tests
> smoke tests
> regression tests
> performance tests
> scalability tests
> resilience tests
>
> Each of these is a tag - a build has a tag placed on it which means at
> the end (when a build has all of the tags) it's passed through all
> levels of testing. A build is "published" (ready for internal people)
> when it's passed the first 3, ready for "ship" when it's passed all of
> them.
>
> This is all pie-in-the-sky right now; I'm starting simple with
> something *not* reliant on SSH (which is a fickle mistress), the
> initial nosejob server I am working on uses SimpleXMLRPCServer to
> manage communications. It enqueues jobs as they come in, so you can
> pile up many requests on a given client.
>
> This is glossing over a lot of my other "wants" - adding plugins for
> nose to support non-python tests, common results formatting, and so
> on. My goals for an initial job-taker (nosejobs) are simple for now.
>
> And if py.execnet/py.test/any other open source work can save me time:
> I plan on absconding with it/using it. I'm not a fan of wheel
> recreation. Yes, I have looked at alternatives - my goal is to make
> this as automatic as possible, support multiple languages for tests,
> and *not* require that a test be written *for* the framework. A test
> should be as simple as a shell script, or as complex as you want, it
> should not have to know about the framework running it.
>
> -jesse
>

Oh yeah, and it will do test inventory-ing for you too. Tired of doing
that by hand.