[TIP] Test isolation // Detection of offending test

Robert Collins robertc at robertcollins.net
Wed Dec 5 16:55:33 PST 2012

On Thu, Dec 6, 2012 at 9:55 AM, Benji York <benji at benjiyork.com> wrote:
> On Wed, Dec 5, 2012 at 4:50 PM, Andres Riancho <andres.riancho at gmail.com> wrote:
>> Lists,
>>     I've got a project with 500+ tests, and during the last month or
>> so I started to notice that some of my tests run perfectly if I run
>> them directly (nosetests --config=nose.cfg
>> core/data/url/tests/test_xurllib.py) but fail when I run them together
>> with all the other tests (nosetests --config=nose.cfg core/).
> [snip]
>>     I suspect that this is a common issue when testing, how do you
>> guys solve this?
> The way I have handled it is to use a test runner that can randomize
> test order and use that option with a continuous integration system like
> Buildbot or Jenkins.  It is especially nice if the test runner tells you
> the seed it used for the random number generator so you can replicate
> the test order yourself.

Oh hai :).

You can also (for a specific case) bisect, if you have a consistent
run order, by running the last 1/2 leading to the failure. Then the
last 1/4 etc until it fails, and when it fails, remove the last 1/N-1
tests (keeping the one that breaks), and so on and so forth.

>>     for test in all_tests:
>>         result = run_nosetests(test, test_that_fails)
>>         if result == False:
>>             print test, 'breaks', test_that_fails
>>     Is that a good idea?
> This is certainly a reasonable way to find the current bad actors, but
> you will need something to keep the pressure on so new ones aren't
> created.

Yeah. Openstack is running through this pain at the moment, as they
bring in testr based parallelisation - lots of brain damaged cruft
turning up and getting fixed.

Andres - if your tests don't use nose specific features, you could use
testtools/subunit to do automation around this - "python -m
testtools.run discover . --load-list FOO" will load a list of test
id's from FOO and run it.


Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Cloud Services

More information about the testing-in-python mailing list