[TIP] Randomizing test order with nose
fuzzyman at voidspace.org.uk
Thu Apr 23 06:25:24 PDT 2009
Douglas Philips wrote:
> On or about 2009 Apr 23, at 8:49 AM, Michael Foord indited:
>> This means that the exact sequence of tests run on any one machine
>> depend on how many computers are running the tests and timing etc.
> Yes, that is a good thing, making sure that test success is not
> accidentally dependent on a specific sequence of prior test runs. We
> have a farm of buildbot slaves, and they run the full suite of tests
> on different devices (which means some tests run only when the right
> device is present) with a shuffled test ordering.
>> Anyway, it shows an alternative approach; rather than making the
>> test order inherently repeatable you could record the order that
>> tests are actually run in and if you *need* to repeat them have a
>> runner capable of doing this. Perhaps not ideal if it is a common
>> need, but combined with our reporting tools it works well for us.
> Our version of unit test logs each test loaded (and even if it was
> loaded from .py vs. .pyc) and the exact sequence that tests were run
> in. We started with that kind of "whole regression suite
> reproducibility". Once we crossed the threshold of several hundred
> tests, including stress tests that can take multiple hours, if not
> days, to run, we needed a way to pluck out one failing test from that
> regression suite and run it in isolation; that can't be done if the
> only way to reproduce the PRNG state for a given test is to re-run the
> entire regression suite, in the same order, up to that test.
What I was actually suggesting didn't involve the use of a PRNG,
although that might not work for every situation. Even if a PRNG is used
to generate the randomized order, if you record the order a PRNG need
not be involved to reproduce it if an alternative runner can get the
precise order from the logs of the previous run.
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
More information about the testing-in-python