[TIP] Branch report and subprocess coverage using Ned's coverage.py in Hudson matrix build job

[Ned Batchelder]

  Hi Olemis,

On 9/21/2010 11:57 AM, Olemis Lang wrote:
> Hi !
>
> I come back once again with the subject of coverage + CI with Hudson.
>
> I run Ned's coverage with `--branch` option set . After generating
> Cobertura XML report it includes tags similar the one shown below :
>
The XML report didn't have correct branch info until the latest release 
(3.4).  Update your coverage.py, and it should work for you.
> So I suppose it's tracking branch coverage. Nonetheless the HTML report
> published by Hudson displays (under Conditionals) `N/A` and `0/0`
> rather than meaningful results.
>
> Q:
>    - Is it the expected behavior or probably I'm missing something?
>    - AYCS `method` tag is always empty. Is it Ok ? Is there a way
>      to make coverage output method info ?
Coverage.py doesn't associate data with methods or classes, only lines 
and files.
> OTOH I'm facing some problems setting up subprocess coverage . I
> read the instructions mentioned in the tutorial [1]_ and tried to
> setup something like this (assume there's a virtualenv under
> $WORKSPACE/venv/$BUILD_NUMBER/ folder, and coverage is already
> installed in there) :
>
> {{{
> #!/bin/bash
>
> echo "**********************************************************"
> echo "* Preparing for subprocess coverage "
> echo "**********************************************************"
>
> echo "import coverage; coverage.process_startup()">
> $WORKSPACE/venv/$BUILD_NUMBER/lib/python$PYTHON/site-packages/sitecustomize.py
> echo "[run]">   .coveragerc
> echo "branch=true">>  .coveragerc
> export COVERAGE_PROCESS_START=.coveragerc
>
> echo "**********************************************************"
> echo "* Running tests with coverage "
> echo "**********************************************************"
>
> $WORKSPACE/venv/$BUILD_NUMBER/bin/coverage run --branch -p setup.py
> test -r ciutils:junitrunner
>
> echo "**********************************************************"
> echo "* Files produced by coverage "
> echo "**********************************************************"
>
> ls -la | grep coverage
>
> echo "**********************************************************"
> echo "* Combining coverage data "
> echo "**********************************************************"
>
> $WORKSPACE/venv/$BUILD_NUMBER/bin/coverage combine
>
> echo "**********************************************************"
> echo "* Generating Cobertura XML report "
> echo "**********************************************************"
>
> $WORKSPACE/venv/$BUILD_NUMBER/bin/coverage xml
> --omit=$WORKSPACE/venv/$BUILD_NUMBER,[...]
>
> }}}
>
> The output contains this snippet so I suppose -p is working
>
> {{{
> **********************************************************
> * Files produced by coverage
> **********************************************************
> -rw-r--r-- 1 hudson nogroup   52 2010-09-20 23:55 .coverage.mypc.7168.064897
> -rw-r--r-- 1 hudson nogroup   18 2010-09-20 23:53 .coveragerc
> }}}
>
> Nevertheless , considering the fact that there's a single file and two
> processes
> it seems that subprocess doesn't track script execution .
>
> BTW , the subprocess is launched this way (so it might
> not be a subprocess strictly speaking, or is it ? )
>
> {{{
> #!py
>
> server = Popen([exe, os.path.join(self.trac_src,
>         'trac', 'web',
>         'standalone.py')] + options + [self.tracdir],
>         stdout=logfile, stderr=logfile,
>         close_fds=close_fds,
>         cwd=self.command_cwd,
>        )
> }}}
>
> In that case exe should be set to
> `sys.executable` so I suppose virtualenv
> sets this value to
> $WORKSPACE/venv/$BUILD_NUMBER/bin/python$PYTHON
> isn't it ?
> The fact is that I can't do anything about that
> since that code is inside Trac source code (in
> `trac.tests.functional.testenv` to be more
> precise ;o).
>
> Q:
>    - What could I do ?
You should verify that the subprocess run by trac is using the Python 
installation and Python path that you've modified.  You also need to 
verify that it sees the COVERAGE_PROCESS_START environment variable.  
Can you modify standalone.py just to get some debugging information from 
it?  Have it dump out the sys.executable, sys,path, and os.environ to 
see what it sees.

You probably want to add "parallel=True" to your .coveragerc, though I 
don't think that's the problem yet.

> Any help will be appreciated . Thnx in advance !
>
> .. [1] Measuring subprocesses
>          (http://nedbatchelder.com/code/coverage/subprocess.html)
>