[TIP] Randomizing test order with nose

[Michael Foord]

Douglas Philips wrote:
> On or about 2009 Apr 22, at 6:36 PM, Scott David Daniels indited:
>   
>> There, the simple expedient of subclassing Random by attaching a  
>> counter to the root
>> pseudo-random number generator meant that you could characterize your
>> position in the random number sequence as (_seed, _calls).
>>     
> ...
>   
>> I think jumpahead is going away altogether in Python 3.X.
>>     
>
> We didn't use jumpahead specifically, but in previous versions of our  
> attempts to make the PRNG reproducible we used a counter (and manually  
> called random() to catch up). It was very tricky to make sure everyone  
> had pulled in the right version of the instrumented random. Had we  
> planned for that up front we could have forced the bottleneck  
> necessary. Instead we've found it was just must simpler to seed the  
> PRNG to captured and known value, when needed.
> Test reproducibility is crucial when debugging large systems. I'm not  
> sure it is of much, if any, use for strict unit-test scope testing.
>   

We *effectively* have test randomization in our distributed system. 
Tests are run on several client computers simultaneously and pull in new 
tests from the database as they execute them.

This means that the exact sequence of tests run on any one machine 
depend on how many computers are running the tests and timing etc.

As the tests are run the results are pushed into a database - and we 
have a tool that can introspect the database and reproduce the exact 
ordering of tests run on any one machine. Useful for trying to reproduce 
failures that only occur when tests are run in certain combinations 
(usually where individual tests manipulate global state and don't 
correctly restore it on completion).

Anyway, it shows an alternative approach; rather than making the test 
order inherently repeatable you could record the order that tests are 
actually run in and if you *need* to repeat them have a runner capable 
of doing this. Perhaps not ideal if it is a common need, but combined 
with our reporting tools it works well for us.

Michael

> -Doug
>
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog