[TIP] Make tox and coverage play together more nicely?

[Marc Schlaich]

tl;dr: Coverage with tox works best and usually as expected by setting
usedevelop = True.

There is even a really simple way to combine various tox runs into one
coverage result:

- As said set `usedevelop = True`. Otherwise the coverage data will be
collected in .tox/{envname}/site-packages and data can't be merged.
- Point COVERAGE_FILE to a specific file for each tox env (e.g.
`.coverage.{envname}`)
- Have a special tox env which is run at the end to combine and report
results, COVERAGE_FILE should point to `.coverage` here
- Optional, but recommended: run `coverage erase` in a first tox env (with
`COVERAGE_FILE=.coverage`, too)

This will be supported by pytest-cov in 2.0 (in development). But in
general this should work with any test runner. pytest-cov doesn't work
right now because it hard codes the data file so setting COVERAGE_FILE is
useless.

See a full example project at
https://github.com/schlamar/pytest-cov/tree/2.0/example-tox-project.

On Fri, Nov 21, 2014 at 6:22 PM, Barry Warsaw <barry at python.org> wrote:

> On Nov 16, 2014, at 12:16 PM, Ned Batchelder wrote:
>
> >If I understand you correctly, this would require coverage to
> automatically
> >understand that .../site-packages/mailman is your own code.  I don't see
> how
> >it could do that.
>
> Yes, it would have to know that `foo` and
> .tox/{envname}/lib/pythonX.Y/site-packages/foo were the same code
>
> It would be nice if 'pythonX.Y' were automatically detected, but I don't
> think
> that's a variable directly available from tox, so it would have to be
> calculated, probably from {envpython}.
>
> >> I also don't like having to explicitly omit everything else in the tox
> >> venv's site-packages.  It seems like since coverage already omits stuff
> in
> >> the system's site-packages, it could do the same with the tox venv
> >> site-packages.
> >
> >Usually the best way to omit things from site-packages is to use the
> >--source=. (or some other value) setting.  In your case, it looks like it
> >would be:
> >
> >     [run]
> >     source =
> >         .
> >         .tox/coverage/lib/python*/site-packages/mailman
> >
> >Would that work?
>
> I'm not sure!  I definitely get different coverage numbers, with more holes
> than I'd expect, but I am going to have to spend more time to investigate
> whether they are legitimate coverage misses or not.  Still, thanks for the
> recommendation!
>
> Cheers,
> -Barry
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20141124/13af4e97/attachment.html>