[TIP] [Python-Dev] Possible language summit topic: buildbots

[C. Titus Brown]

On Sun, Oct 25, 2009 at 08:54:46AM +0000, Mark Dickinson wrote:
> Would it be worth spending some time discussing the buildbot situation
> at the PyCon 2010 language summit?  In the past, I've found the
> buildbots to be an incredibly valuable resource;  especially when
> working with aspects of Python or C that tend to vary significantly
> from platform to platform (for me, this usually means floating-point,
> and platform math libraries, but there are surely many other things it
> applies to).  But more recently there seem to have been some
> difficulties keeping a reasonable number of buildbots up and running.
> A secondary problem is that it can be awkward to debug some of the
> more obscure test failures on buildbots without having direct access
> to the machine.  From conversations on IRC, I don't think I'm alone in
> wanting to find ways to make the buildbots more useful.
> 
> So the question is: how best to invest time and possibly money to
> improve the buildbot situation (and as a result, I hope, improve the
> quality of Python)?  What could be done to make maintenance of build
> slaves easier?  Or to encourage interested third parties to donate
> hardware and time?  Are there good alternatives to Buildbot that might
> make a difference? What do other projects do?
> 
> These are probably the wrong questions;  I'm hoping that a discussion
> would help produce the right questions, and possibly some answers.

[ x-posting to testing-in-python; please redirect followups to one list or
the other! ]

Hi Mark,

a few bits of information...

---

I have a set of VM machines running some "core" build archs -- Linux, Mac OS X,
Win XP, Win Vista, and Win 7 -- that I am going to dedicate to this purpose.
I am happy to give out remote admin access to a few people.  This
infrastructure is also going to increase slowly as I build up my lab's internal
network.

I'm giving Tarek an account on my Linux box later today to serve as a build
slave for Distribute.

--

More machines, and more esoteric machines, will be coming online as Snakebite
unfolds.  Trent Nelson (Snakepit master) has been finishing up some consulting
work and is going to dedicate his time to it starting in November.  This means
more 64 bit, bigmem, and "weird" archs, with full login access.

---

I've also been working on a buildbot alternative that I'm calling pony-build.
pony-build is based on a client-push architecture in which client machines
do builds and push results to the master, which then acts as a record-keeper
rather than a slave driver.  The result is a less coordinated but (AFAICT) much
less fragile continuous integration system.  I'm hoping to give a talk at PyCon
on the differences, and there will be a sprint on pony-build + pyhton-dev at
PyCon, regardless.

The current status of pony-build is "functional but ugly inside".  In
particular, the data model is horrible, and the internal API needs much
more fleshing out.  Nonetheless, my server has been running for two months
or so, and you can look at the results here,

	http://lyorn.idyll.org/ctb/pb-dev/

The most fully-fleshed out set of build clients is for pygr,

	http://lyorn.idyll.org/ctb/pb-dev/pygr/

but you can view daily build results for Python 2.7 at

	http://lyorn.idyll.org/ctb/pb-dev/python/

with an exhaustive list here

	http://lyorn.idyll.org/ctb/pb-dev/python/show_all

(and why the heck am I sorting in reverse date order, anyway?!)

The most interesting (?) part of pony-build right now is the client
config, which I'm struggling to make simple and potentially universal
enough to serve under buildbot as well:

	http://github.com/ctb/pony-build/blob/master/client/build-cpython

(see 'commands' list).

The most *exciting* part of pony-build, apart from the always-riveting
spectacle of "titus rediscovering problems that buildbot solved 5 years ago",
is the loose coupling of recording server to the build slaves and build
reporters.  My plan is to enable a simple and lightweight XML-RPC and/or
REST-ish interface for querying the recording server from scripts or other Web
sites.  This has Brett aquiver with anticipation, I gather -- no more visual
inspection of buildbot waterfall pages ;)

--

If you're interested in bashing on, contributing to, or helping figure out
what color the pony-build main screen should be, contact me off-list; I'm
reluctant to spam up python-dev or testing-in-python.

That having been said, the results of taking it and trying it out -- you can
post results to my own recording server at 

	http://lyorn.idyll.org/ctb/pb-dev/xmlrpc

-- would be most welcome.

Once I fix the data model, code collaboration will be much more feasible,
too.

---

cheers,
--titus
-- 
C. Titus Brown, ctb at msu.edu