[TIP] Randomizing test order with nose

[Scott David Daniels]

Douglas Philips wrote:
 > On 2009 Apr 22, at 12:51 PM, Scott David Daniels indited:
 >> Be careful here.  For randomized tests in any form (including 
 >> execution order), it is vital to be able to rerun _exactly_the_
 >> _same_test(s)_.  So, provide a command line arg (or whatever)
 >> that allows you to seed the random number generator, and always
 >> provide the seed used when running a randomized system.
 >
 > Yes!
 > We've found that it is crucial to be able to reproduce test runs.
 > Even more crucial is to be able to pluck out a failing test and rerun 
 > it with the exact same random-number-environment.
 > Our solution is:
 >      provide a command line argument (to support "reproducing results")
 >      have the framework itself print out the seed used for each run 
 > (to maximize the shuffling and still allow reproducible results)
 >      have the framework itself reset the seed to the known value 
 > before each test (to enable re-running of just one test)
 >
 > I'm curious to know if there is a simpler way to achieve those three 
 > goals.

Well, the answer is changing as we move through Python versions, and
not (for my money) for the better.  Once upon a time, the random number
generator supported a call, "jumpahead(n)" which moved forward n elements.
In the pre-2.3 world, "jumpahead(n)" moved forward n elements.  There, the
simple expedient of subclassing Random by attaching a counter to the root
pseudo-random number generator meant that you could characterize your
position in the random number sequence as (_seed, _calls).  Once you
did that, you can return to a point in the call sequence by calling
    rand = CountingRandom()
    rand.seed(_seed)
    rand.jumpahead(_calls)
At 2.3, .jumpahead(n) was redefined as "Change the internal state to one
different from and likely far away from the current state."  Not incredibly
useful for my purposes (though useful for some RNG uses).  I think jumpahead
is going away altogether in Python 3.X.
So, I add .tell and .seek methods to my Random subclass, (treating the RNG
as if it were a kind of file of randomization), and then as part up setUp
you can save the results of .tell, and on any failure display that value,
thus allowing you to specify (_seed, _calls) when retrying.

Your solution looks pretty good, though.

p.s.  Sorry about the lines before, Thunderbird used to be a lot better at
letting me see when I had a long line, but it now it fiddles e-mail fonts
on me; I don't always guess correctly about long lines.

--Scott David Daniels
Scott.Daniels at Acm.Org