[TIP] coverage combine with the results of a jenkins matrix job...

[Chris Withers]

If anyone's interested, here's the successful config:

https://github.com/Simplistix/testfixtures/blob/py3k/.coveragerc

I then do:

coverage combine
coverage html -i
coverage xml -i

cheers,

Chris

On 18/02/2013 01:40, Ned Batchelder wrote:
> Chris,
>
> It's a bit hard to see what's going on with all these reports, etc, but
> perhaps this will help:  Paths in the [paths] configuration section are
> meant to be the root of the tree.  That's the reason for the prohibition
> on ending with a wildcard.  Remove those stars at the end, and you might
> get the results you want.
>
> We can take the rest of the conversation off-list.
>
> --Ned.
>
> On 2/16/2013 4:33 AM, Chris Withers wrote:
>> Hi Ned,
>>
>> I'm all at sea on this, still.
>>
>> I added the "parallel = True" option to my .coveragerc and now appear
>> to have all the coverage files as expected:
>>
>> http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/PYTHON=3.3,label=linux/
>>
>> ...etc...
>> http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/PYTHON=2.6,label=windows/
>>
>>
>> Here's how I tried to combine:
>>
>> $ cp */.coverage.* .
>> $ coverage combine
>>
>> ...using:
>>
>> http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/.coveragerc/*view*/
>>
>>
>> ...but 'coverage report' gives:
>>
>> http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/report.txt/*view*/
>>
>>
>> What do these lines mean:
>>
>> '/var/lib/jenkins/slave/workspace/testfixtures-buildout/4326993d/testfixtures/tests/configparser-read.txt:24':
>> [Errno 2] No such file or directory:
>> '/var/lib/jenkins/slave/workspace/testfixtures-buildout/4326993d/testfixtures/tests/configparser-read.txt:24'
>>
>>
>> Also, I need to get around hard-coding those last segments, as they
>> may change on each run, so I tried the above process with a
>> .coveragerc as here:
>>
>> http://jenkins.simplistix.co.uk/job/testfixtures-coverage/ws/.coveragerc.wrong/*view*/
>>
>>
>> ...but got:
>>
>> $ coverage combine --rcfile=.coveragerc.wrong
>> Pattern must not end with wildcards.
>>
>> Not sure what I should do here? The wildcard bit *is* the last part in
>> the path, and the testfixtures package lives underneath that:
>>
>> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=3.3,label=linux/ws/
>>
>>
>> Also confused by this:
>>
>> (d41d8cd9)jenkins at server2:~/slave/workspace/testfixtures-coverage$
>> coverage combine
>> (d41d8cd9)jenkins at server2:~/slave/workspace/testfixtures-coverage$
>> coverage report testfixtures
>> Name           Stmts   Miss Branch BrMiss  Cover
>> ------------------------------------------------
>> testfixtures   NoSource: No source for code:
>> '/var/lib/jenkins/slave/workspace/testfixtures-coverage/testfixtures':
>> [Errno 21] Is a directory:
>> '/var/lib/jenkins/slave/workspace/testfixtures-coverage/testfixtures'
>>
>> I don't understand, 'testfixtures' is a module name and that's what
>> 'coverage report' says it wants:
>>
>> $ coverage report --help
>> Usage: coverage report [options] [modules]
>>
>> Report coverage statistics on modules.
>> ....
>>
>> Anyway, any help very gratefully appreciated :-)
>>
>> Chris
>>
>> Here's the .coveragec
>> On 15/02/2013 03:18, Ned Batchelder wrote:
>>> Chris,
>>>
>>> If you use the --parallel flag, the .coverage files will be given
>>> distinct names.  You can then copy them to a single directory, and
>>> "coverage combine" will combine them.  You'll need a [path] section if
>>> the same source file is found at different paths for different runs.
>>>
>>> --Ned.
>>>
>>> On 2/14/2013 6:25 PM, Chris Withers wrote:
>>>> (copying the list, since I think this may be of wider interest)
>>>>
>>>> Hi Ned,
>>>>
>>>> Okay, I think I'm part of the way there, but still confused, hopefully
>>>> you can help...
>>>>
>>>> So, the runs generating the .coverage files are in this matrix:
>>>>
>>>> http://jenkins.simplistix.co.uk/view/testfixtures/job/testfixtures-buildout/
>>>>
>>>>
>>>>
>>>> Here's an example output:
>>>>
>>>> http://jenkins.simplistix.co.uk/view/testfixtures/job/testfixtures-buildout/PYTHON=3.3,label=linux/182/console
>>>>
>>>>
>>>>
>>>> The actual command being run is:
>>>>
>>>> bin/nosetests --with-xunit --with-cov --cov=testfixtures
>>>> --cov-report=xml --cov-report=html
>>>>
>>>> So, now I have a downstream job where I want to combine all those
>>>> .coverage files and generate html and xml reports. Here's the
>>>> workspace for that job:
>>>>
>>>> http://jenkins.simplistix.co.uk/view/testfixtures/job/testfixtures-coverage/ws/
>>>>
>>>>
>>>>
>>>> So, each PYTHON* has a .coverage file in it, testfixtures-3.0.0dev
>>>> contains the source.
>>>>
>>>> I thought it would just be a case of:
>>>>
>>>> coverage combine PYTHON*/.coverage
>>>> coverage xml
>>>> coverage html
>>>>
>>>> But combine doesn't appear to work as I expect. In fact, I don't think
>>>> I have a clue how it's supposed to work, so thought I'd ask ;-)
>>>> What should I be looking to do? I have a feeling I'll need a [path]
>>>> section in it, but no idea what I should put there...
>>>>
>>>> Any help very gratefully received!
>>>>
>>>> Chris
>>>>
>>>> On 13/02/2013 16:21, Ned Batchelder wrote:
>>>>> Sorry, I'm not well-versed enough with Jenkins to know.
>>>>>
>>>>> --Ned.
>>>>>
>>>>> On 2/13/2013 10:57 AM, Chris Withers wrote:
>>>>>> Hmm, any idea how to do that in a Jenkins matrix job?
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>> On 13/02/2013 15:52, Ned Batchelder wrote:
>>>>>>> I would run coverage twice: for Python 2, and again for Python 3,
>>>>>>> and
>>>>>>> then I would combine the results (coverage combine).
>>>>>>>
>>>>>>> --Ned.
>>>>>>>
>>>>>>> On 2/13/2013 10:36 AM, Chris Withers wrote:
>>>>>>>> One question: if I have code paths that should not be covered
>>>>>>>> under a
>>>>>>>> particular version of python, but should under others, how can I go
>>>>>>>> about dealing with that?
>>>>>>>>
>>>>>>>> (eg:
>>>>>>>>
>>>>>>>> if PY3:
>>>>>>>> ...
>>>>>>>> else:
>>>>>>>> ...
>>>>>>>>
>>>>>>>> both of those branches should be covered in different test runs,
>>>>>>>> but
>>>>>>>> neither of them will be covered in a single run.
>>>>>>>>
>>>>>>>> How would you tackle this to give meaningful output?
>>>>>>>>
>>>>>>>> cheers,
>>>>>>>>
>>>>>>>> Chris
>>>>
>>>
>>>
>>> ______________________________________________________________________
>>> This email has been scanned by the Symantec Email Security.cloud
>>> service.
>>> For more information please visit http://www.symanteccloud.com
>>> ______________________________________________________________________
>>
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk