[TIP] coverage combine with the results of a jenkins matrix job...

[Ned Batchelder]

Chris,

It's a bit hard to see what's going on with all these reports, etc, but 
perhaps this will help:  Paths in the [paths] configuration section are 
meant to be the root of the tree.  That's the reason for the prohibition 
on ending with a wildcard.  Remove those stars at the end, and you might 
get the results you want.

We can take the rest of the conversation off-list.

--Ned.

On 2/16/2013 4:33 AM, Chris Withers wrote:
> Hi Ned,
>
> I'm all at sea on this, still.
>
> I added the "parallel = True" option to my .coveragerc and now appear 
> to have all the coverage files as expected:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/PYTHON=3.3,label=linux/ 
>
> ...etc...
> http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/PYTHON=2.6,label=windows/ 
>
>
> Here's how I tried to combine:
>
> $ cp */.coverage.* .
> $ coverage combine
>
> ...using:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/.coveragerc/*view*/ 
>
>
> ...but 'coverage report' gives:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/report.txt/*view*/ 
>
>
> What do these lines mean:
>
> '/var/lib/jenkins/slave/workspace/testfixtures-buildout/4326993d/testfixtures/tests/configparser-read.txt:24': 
> [Errno 2] No such file or directory: 
> '/var/lib/jenkins/slave/workspace/testfixtures-buildout/4326993d/testfixtures/tests/configparser-read.txt:24'
>
> Also, I need to get around hard-coding those last segments, as they 
> may change on each run, so I tried the above process with a 
> .coveragerc as here:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/.coveragerc.wrong/*view*/ 
>
>
> ...but got:
>
> $ coverage combine --rcfile=.coveragerc.wrong
> Pattern must not end with wildcards.
>
> Not sure what I should do here? The wildcard bit *is* the last part in 
> the path, and the testfixtures package lives underneath that:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=3.3,label=linux/ws/ 
>
>
> Also confused by this:
>
> (d41d8cd9)jenkins at server2:~/slave/workspace/testfixtures-coverage$ 
> coverage combine
> (d41d8cd9)jenkins at server2:~/slave/workspace/testfixtures-coverage$ 
> coverage report testfixtures
> Name           Stmts   Miss Branch BrMiss  Cover
> ------------------------------------------------
> testfixtures   NoSource: No source for code: 
> '/var/lib/jenkins/slave/workspace/testfixtures-coverage/testfixtures': 
> [Errno 21] Is a directory: 
> '/var/lib/jenkins/slave/workspace/testfixtures-coverage/testfixtures'
>
> I don't understand, 'testfixtures' is a module name and that's what 
> 'coverage report' says it wants:
>
> $ coverage report --help
> Usage: coverage report [options] [modules]
>
> Report coverage statistics on modules.
> ....
>
> Anyway, any help very gratefully appreciated :-)
>
> Chris
>
> Here's the .coveragec
> On 15/02/2013 03:18, Ned Batchelder wrote:
>> Chris,
>>
>> If you use the --parallel flag, the .coverage files will be given
>> distinct names.  You can then copy them to a single directory, and
>> "coverage combine" will combine them.  You'll need a [path] section if
>> the same source file is found at different paths for different runs.
>>
>> --Ned.
>>
>> On 2/14/2013 6:25 PM, Chris Withers wrote:
>>> (copying the list, since I think this may be of wider interest)
>>>
>>> Hi Ned,
>>>
>>> Okay, I think I'm part of the way there, but still confused, hopefully
>>> you can help...
>>>
>>> So, the runs generating the .coverage files are in this matrix:
>>>
>>> http://jenkins.simplistix.co.uk/view/testfixtures/job/testfixtures-buildout/ 
>>>
>>>
>>>
>>> Here's an example output:
>>>
>>> http://jenkins.simplistix.co.uk/view/testfixtures/job/testfixtures-buildout/PYTHON=3.3,label=linux/182/console 
>>>
>>>
>>>
>>> The actual command being run is:
>>>
>>> bin/nosetests --with-xunit --with-cov --cov=testfixtures
>>> --cov-report=xml --cov-report=html
>>>
>>> So, now I have a downstream job where I want to combine all those
>>> .coverage files and generate html and xml reports. Here's the
>>> workspace for that job:
>>>
>>> http://jenkins.simplistix.co.uk/view/testfixtures/job/testfixtures-coverage/ws/ 
>>>
>>>
>>>
>>> So, each PYTHON* has a .coverage file in it, testfixtures-3.0.0dev
>>> contains the source.
>>>
>>> I thought it would just be a case of:
>>>
>>> coverage combine PYTHON*/.coverage
>>> coverage xml
>>> coverage html
>>>
>>> But combine doesn't appear to work as I expect. In fact, I don't think
>>> I have a clue how it's supposed to work, so thought I'd ask ;-)
>>> What should I be looking to do? I have a feeling I'll need a [path]
>>> section in it, but no idea what I should put there...
>>>
>>> Any help very gratefully received!
>>>
>>> Chris
>>>
>>> On 13/02/2013 16:21, Ned Batchelder wrote:
>>>> Sorry, I'm not well-versed enough with Jenkins to know.
>>>>
>>>> --Ned.
>>>>
>>>> On 2/13/2013 10:57 AM, Chris Withers wrote:
>>>>> Hmm, any idea how to do that in a Jenkins matrix job?
>>>>>
>>>>> Chris
>>>>>
>>>>> On 13/02/2013 15:52, Ned Batchelder wrote:
>>>>>> I would run coverage twice: for Python 2, and again for Python 3, 
>>>>>> and
>>>>>> then I would combine the results (coverage combine).
>>>>>>
>>>>>> --Ned.
>>>>>>
>>>>>> On 2/13/2013 10:36 AM, Chris Withers wrote:
>>>>>>> One question: if I have code paths that should not be covered 
>>>>>>> under a
>>>>>>> particular version of python, but should under others, how can I go
>>>>>>> about dealing with that?
>>>>>>>
>>>>>>> (eg:
>>>>>>>
>>>>>>> if PY3:
>>>>>>> ...
>>>>>>> else:
>>>>>>> ...
>>>>>>>
>>>>>>> both of those branches should be covered in different test runs, 
>>>>>>> but
>>>>>>> neither of them will be covered in a single run.
>>>>>>>
>>>>>>> How would you tackle this to give meaningful output?
>>>>>>>
>>>>>>> cheers,
>>>>>>>
>>>>>>> Chris
>>>
>>
>>
>> ______________________________________________________________________
>> This email has been scanned by the Symantec Email Security.cloud 
>> service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________________________________________________
>