[TIP] Unittest framework with good process isolation

[holger krekel]

Hi Jonathan,

On Fri, Aug 03, 2012 at 14:31 -0700, Jonathan Hunt wrote:
> Hi,
> 
> We have written our tests using the built-in python unittest framework and
> we currently use nosetest to collect, run and report the tests.
> 
> Things we like about nose:
> 
>    - uses standard python unit tests (we like the structure this imposes).
>    - supports reporting coverage and test output in xml (for jenkins).
> 
> What we are missing is a good way to run tests in isolated processes while
> maintaining good error repoorting (we are testing C++ libraries through
> python so segfaults should not be catastrophic and each unittest is
> designed to run in isolation). nosepipe seems to be no longer maintained
> and we have some problems with it.
>
> We are trying to figure out whether we should - fix/use nosepipe - switch
> to nose2 and write nosepipe2. - use pytest or some other testing framework.

pytest has the "xdist" plugin (http://pypi.python.org/pypi/pytest-xdist ) which

a) provides the --boxed option to run each test in subprocess which allows
   to survive and report on crashes of single tests
   
b) provides the -n NUM option to create parallel execution to speed up
   your test run.  
   
I've added examples here: http://stackoverflow.com/a/11806997/137901

As to coverage, you may use the pytest-cov plugin which also supports
parallel execution: http://pypi.python.org/pypi/pytest-cov/

Generation of xml reports to a path are supported via the builtin plugin
and the ``--junitxml=path`` option. 

As pytest also supports the running of python-unittest based tests
you can use all of these features out of the box for them as well.

If you are willing or interested, you can write some additional
tests without unittest.TestCase subclassing, giving you otherwise
not available parametrization and functional testing features, see
http://pytest.org/latest/funcargs.html.

best,

holger

> We would prefer to use an approach with a good community. It seems our
> problem (C++ plugins requiring good isolation) might be a common problem
> but googling I have not found solutions that are maintained. Advice from
> more experienced heads appreciated.


> Regards,
> Jonny

> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python