[TIP] Test isolation // Detection of offending test

[Andres Riancho]

Lists,

    I've got a project with 500+ tests, and during the last month or
so I started to notice that some of my tests run perfectly if I run
them directly (nosetests --config=nose.cfg
core/data/url/tests/test_xurllib.py) but fail when I run them together
with all the other tests (nosetests --config=nose.cfg core/).

    I've tried nosetest's --with-isolation without much luck: it
breaks more than it fixes.

    Given that the project has 500+ tests, it is very difficult for me
to identify exactly which one is affecting the tests that fail.

    I suspect that this is a common issue when testing, how do you
guys solve this? Is there a way to detect which plugin is not properly
isolated? I thought about writing a script that would do something
like this:

    for test in all_tests:
        result = run_nosetests(test, test_that_fails)
        if result == False:
            print test, 'breaks', test_that_fails

    Is that a good idea?

PS: Sorry for the cross-post between the lists but I thought it was
important for both

Regards,
--
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3