[TIP] Nosejobs!

[Jesse Noller]

On Fri, Apr 10, 2009 at 3:21 PM, Doug Philips <dgou at mac.com> wrote:
> On or about Friday, April 10, 2009, at 02:12PM, Jesse Noller indited:
>>Facelift provisions from pre-built images; those images have certain
>>resources already online, such as the daemon, python and anything else
>>I need and can pre-allocate on the OS. I've done this before with PXE
>>booting, and it worked quite well.
>
> Ugh, that is way more heavy weight than something simple. Also PXE booting, while simplistic, is hardly simple either, esp for a vastly distributed set up where you might not have complete control. That is one place that buildbot rocks. Complete control over the slave host is one kind of simplicity. Not needing that kind of control is a more powerful form of simplicity.
>

You can support multiple methods of managing and provisioning slaves.

>
>
>>Clients report back (or can be polled) for results encapsulated in
>>JSON or YAML (since it's cross language and stupid simple).
>
> What's your opinion on the Pandokia format? Do you need the nesting that JSON provides? If JSON isn't "for free" in the standard python library, I'm not sure that extra dependency is worth it over the Pandokia format's simplicity...
>
>

JSON is in python2.6

Also, I've already built things using "simple" formats - plain
INI-style formats, etc. I found they didn't scale and were not
extensible enough for me long term.

>
>>All communication is secure (SSL) as this is intended to be a highly
>>distributed system.
>
> ?? I thought in other threads you were saying that SSL was way too fragile? Must be on email memory overload.
>
>

What? No, SSH. SSL != SSH.

>
>>If resources are unavailable, the client-server (nosejobs) reports
>>such, and it can be dealt with. If something goes out for tacos, we
>>deal with it.
>
> Oh the lies begin... :) "deal with it" hides a world of complexity hurt.
> You've got the firewood under the pot holding the frog... (see below for more on the FUD :) )
>

Please. I've outlining high level concepts, not giving you an
engineering spec. I know full well what needs to go into this.


>>A problem like this shares many attributes with distributed systems -
>>you have to be able to withstand faults, transient issues, homogeny on
>>the network and so on.
>
> Quite so. What you are talking about is remote job management.
> That the jobs "happen to be tests" is irrelevant.

Yup. I have some reasoning to my madness.

>>I'm interested in the FUD you speak of, but I agree: I can't be
>>screwed trying to grok something with abysmal documentation, or
>>confusing and sadness inducing code.
>
> Indeed. Look at the scorn that Zope or Plone gets. I was meta-amused, but saddened, by the
> Zope lightning talk:
> '"oooh, couch DB" - we've been there, done that for a long time.'
>

Yeah. that's a different issue. Oh god don't get me started.

> Attended a Jython Open Space Thursday before #pycon started. They were lamenting that most
> folks thought Jython was dead because the front page of the web site hadn't been updated in
> forever... (I believe they've started to fix that now).
>

until they get the pending release done and update the docs; it is dead.

>>No, what you hear me saying is "I've tried all of this other stuff,
>>and while I might adapt some of it, it doesn't fit my needs, it
>>doesn't scale, it's obtuse, broken, or non applicable".
>
> Fair enough, I'm new to this list, and if this is old ground, my apologies.

Nope, just old ground for some of us. And I'm not trying to bite your
head off, again - if I find something which I can abscond with/use, I
will. I have no intention to recreate the world.

> Pandokia's spec seems awful simple.
> Personally, I'm rather fond of Titus' initial proposal: Work out the results level protocol, then build
> tools around that. Yes, you might want to build tools to help work that out, but if you've done lots
> of these systems before, as have some of the rest of us, we should be able to leverage that
> knowledge...
>

My intention is to start small, build out big, but keeping the end in
mind. Nothing you do now should prevent you from expanding/making it
better in the future.


> Fiat away. I might even pitch in if you don't use CVS or SVN :)
> I'll certainly call BS when it see it, if nothing else. :)

Mercurial up in this piece, son!

-jesse