[TIP] Randomizing test order with nose

[Michael Foord]

Douglas Philips wrote:
> On or about 2009 Apr 23, at 8:49 AM, Michael Foord indited:
>   
>> This means that the exact sequence of tests run on any one machine  
>> depend on how many computers are running the tests and timing etc.
>>     
>
> Yes, that is a good thing, making sure that test success is not  
> accidentally dependent on a specific sequence of prior test runs. We  
> have a farm of buildbot slaves, and they run the full suite of tests  
> on different devices (which means some tests run only when the right  
> device is present) with a shuffled test ordering.
>
>   
>> Anyway, it shows an alternative approach; rather than making the  
>> test order inherently repeatable you could record the order that  
>> tests are actually run in and if you *need* to repeat them have a  
>> runner capable of doing this. Perhaps not ideal if it is a common  
>> need, but combined with our reporting tools it works well for us.
>>     
>
> Our version of unit test logs each test loaded (and even if it was  
> loaded from .py vs. .pyc) and the exact sequence that tests were run  
> in. We started with that kind of "whole regression suite  
> reproducibility". Once we crossed the threshold of several hundred  
> tests, including stress tests that can take multiple hours, if not  
> days, to run, we needed a way to pluck out one failing test from that  
> regression suite and run it in isolation; that can't be done if the  
> only way to reproduce the PRNG state for a given test is to re-run the  
> entire regression suite, in the same order, up to that test.
>   

What I was actually suggesting didn't involve the use of a PRNG, 
although that might not work for every situation. Even if a PRNG is used 
to generate the randomized order, if you record the order a PRNG need 
not be involved to reproduce it if an alternative runner can get the 
precise order from the logs of the previous run.

Michael

> -Doug
>
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog