[TIP] alphabetical order in testing

[holger krekel]

On Wed, Jan 20, 2010 at 16:31 -0800, Mark Waite wrote:
> > From: holger krekel <holger at merlinux.eu>
> > To: Mark Sienkiewicz <sienkiew at stsci.edu>
> > Cc: TesttingInPython <testing-in-python at lists.idyll.org>
> > Sent: Wed, January 20, 2010 4:51:47 PM
> > Subject: Re: [TIP] alphabetical order in testing
> 
> > >> I agree that stable ordering is valuable for replicating test failures. 
> > >> Both file-position and alphabetical orderings are stable.    
> > >
> > > Yes.  There are good arguments in favor of each too.  There is also a  
> > > good argument for the ability to choose a random order, as long as you  
> > > can get exactly the same random order again when you want it.
> > 
> > yip, was discussed some time ago on this list IIRC. 
> > 
> 
> I was never able to get test order randomization to work using nose based on the previous thread on this list.  Are there other test order randomization techniques that could be used with nose?  
> 
> Alternately, I'm willing to use another test discovery tool if there are other test runners which will find my tests, run them, report their results in xunit format, and randomize test order.

My offer: you might give py.test's xdist plugin a try, checkout
http://bitbucket.org/hpk42/pytest-xdist 

and declare option(s) in xdist/plugin.py and go to 
xdist/dsession.py::collect_all_items() where you have a full 
list of test items which you can re-order according to your options
which you obtain by "self.config.getvalue('dest_used_in_option_decl')". 
Then issue "python setup.py develop" and run "py.test -v -n1" which should
run the tests in a subprocess in your order.  Feel free to 
mail me personally if you hit issues or have questions. 

A warning note: py.test's collection of unittest.py based tests is probably
not as general as nosetests's one. py.test also runs many tests written 
for nose, but i also wouldn't call that complete.  I am willing help
improve that given repeatable problems with existing test code. 

cheers & best,
holger