[TIP] coverage combine with the results of a jenkins matrix job...

[Chris Withers]

Hi Ned,

I'm all at sea on this, still.

I added the "parallel = True" option to my .coveragerc and now appear to 
have all the coverage files as expected:

http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/PYTHON=3.3,label=linux/
...etc...
http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/PYTHON=2.6,label=windows/

Here's how I tried to combine:

$ cp */.coverage.* .
$ coverage combine

...using:

http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/.coveragerc/*view*/

...but 'coverage report' gives:

http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/report.txt/*view*/

What do these lines mean:

'/var/lib/jenkins/slave/workspace/testfixtures-buildout/4326993d/testfixtures/tests/configparser-read.txt:24': 
[Errno 2] No such file or directory: 
'/var/lib/jenkins/slave/workspace/testfixtures-buildout/4326993d/testfixtures/tests/configparser-read.txt:24'

Also, I need to get around hard-coding those last segments, as they may 
change on each run, so I tried the above process with a .coveragerc as here:

http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/.coveragerc.wrong/*view*/

...but got:

$ coverage combine --rcfile=.coveragerc.wrong
Pattern must not end with wildcards.

Not sure what I should do here? The wildcard bit *is* the last part in 
the path, and the testfixtures package lives underneath that:

http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=3.3,label=linux/ws/

Also confused by this:

(d41d8cd9)jenkins at server2:~/slave/workspace/testfixtures-coverage$ 
coverage combine
(d41d8cd9)jenkins at server2:~/slave/workspace/testfixtures-coverage$ 
coverage report testfixtures
Name           Stmts   Miss Branch BrMiss  Cover
------------------------------------------------
testfixtures   NoSource: No source for code: 
'/var/lib/jenkins/slave/workspace/testfixtures-coverage/testfixtures': 
[Errno 21] Is a directory: 
'/var/lib/jenkins/slave/workspace/testfixtures-coverage/testfixtures'

I don't understand, 'testfixtures' is a module name and that's what 
'coverage report' says it wants:

$ coverage report --help
Usage: coverage report [options] [modules]

Report coverage statistics on modules.
....

Anyway, any help very gratefully appreciated :-)

Chris

Here's the .coveragec
On 15/02/2013 03:18, Ned Batchelder wrote:
> Chris,
>
> If you use the --parallel flag, the .coverage files will be given
> distinct names.  You can then copy them to a single directory, and
> "coverage combine" will combine them.  You'll need a [path] section if
> the same source file is found at different paths for different runs.
>
> --Ned.
>
> On 2/14/2013 6:25 PM, Chris Withers wrote:
>> (copying the list, since I think this may be of wider interest)
>>
>> Hi Ned,
>>
>> Okay, I think I'm part of the way there, but still confused, hopefully
>> you can help...
>>
>> So, the runs generating the .coverage files are in this matrix:
>>
>> http://jenkins.simplistix.co.uk/view/testfixtures/job/testfixtures-buildout/
>>
>>
>> Here's an example output:
>>
>> http://jenkins.simplistix.co.uk/view/testfixtures/job/testfixtures-buildout/PYTHON=3.3,label=linux/182/console
>>
>>
>> The actual command being run is:
>>
>> bin/nosetests --with-xunit --with-cov --cov=testfixtures
>> --cov-report=xml --cov-report=html
>>
>> So, now I have a downstream job where I want to combine all those
>> .coverage files and generate html and xml reports. Here's the
>> workspace for that job:
>>
>> http://jenkins.simplistix.co.uk/view/testfixtures/job/testfixtures-coverage/ws/
>>
>>
>> So, each PYTHON* has a .coverage file in it, testfixtures-3.0.0dev
>> contains the source.
>>
>> I thought it would just be a case of:
>>
>> coverage combine PYTHON*/.coverage
>> coverage xml
>> coverage html
>>
>> But combine doesn't appear to work as I expect. In fact, I don't think
>> I have a clue how it's supposed to work, so thought I'd ask ;-)
>> What should I be looking to do? I have a feeling I'll need a [path]
>> section in it, but no idea what I should put there...
>>
>> Any help very gratefully received!
>>
>> Chris
>>
>> On 13/02/2013 16:21, Ned Batchelder wrote:
>>> Sorry, I'm not well-versed enough with Jenkins to know.
>>>
>>> --Ned.
>>>
>>> On 2/13/2013 10:57 AM, Chris Withers wrote:
>>>> Hmm, any idea how to do that in a Jenkins matrix job?
>>>>
>>>> Chris
>>>>
>>>> On 13/02/2013 15:52, Ned Batchelder wrote:
>>>>> I would run coverage twice: for Python 2, and again for Python 3, and
>>>>> then I would combine the results (coverage combine).
>>>>>
>>>>> --Ned.
>>>>>
>>>>> On 2/13/2013 10:36 AM, Chris Withers wrote:
>>>>>> One question: if I have code paths that should not be covered under a
>>>>>> particular version of python, but should under others, how can I go
>>>>>> about dealing with that?
>>>>>>
>>>>>> (eg:
>>>>>>
>>>>>> if PY3:
>>>>>> ...
>>>>>> else:
>>>>>> ...
>>>>>>
>>>>>> both of those branches should be covered in different test runs, but
>>>>>> neither of them will be covered in a single run.
>>>>>>
>>>>>> How would you tackle this to give meaningful output?
>>>>>>
>>>>>> cheers,
>>>>>>
>>>>>> Chris
>>
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk