[TIP] massive parallel testing in python

Kumar McMillan kumar.mcmillan at gmail.com
Mon Dec 3 11:36:36 PST 2007

On Nov 21, 2007 10:55 AM, Noah Gift <noah.gift at gmail.com> wrote:
> I have another "thing" I am working on where I am interesting in doing
> massive parallel testing as a current series of thousands of tests
> takes over 24 hours.  I am extremely ignorant of open source tools in
> Python that do this already, but one Idea I had was to use parallel
> python and to setup a bunch of testing nodes:
> http://www.parallelpython.com/
> Is there anything pre-built that someone could recommend?

Hi Noah.

This is a very important "thing" you are working on ;)  As all of our
test suites start to grow, the idea of running tests in parallel
becomes more appealing [1].  I have looked at parallel python and
decided it wasn't a good fit as it is designed to optimize a single
function.  However, I was looking at it from the perspective of
building a plugin with it for nose that can run tests in parallel.  I
would urge you to check out the threads on the parallel testing plugin
for nose idea:

...and it would be great if you could try out an implementation.
There are a lot of people interested in using a plugin to run
nosetests in parallel.

Alternatively you could look at testoob
(http://testoob.sourceforge.net/), I'd be curious to see a report from
someone who tried to use its parallel testing features.  Also, maybe
someone on the list can speak about what kind of parallel testing
features py.test offers, as I vaguely recall hearing it supports some,
but I could be making that up.


[1] tests that automate a browser are the slowest and benefit the most
from running in parallel.  At my company we have begun experimenting
with Selenium Grid
(http://www.infoq.com/news/2007/09/selenium-grid-parallel-testing) for

> The Pyvix
> buildbot stuff seems interesting, but I am not sure if that directly
> applies for me.
> Noah
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python

More information about the testing-in-python mailing list