[TIP] Coverage does not cover script if py.test executes it from another directory

[Christoph Buchner]

Ned Batchelder schrieb am 12.09.2013 03:40:

> 
> On 9/8/13 10:23 AM, Christoph Buchner wrote:
>>
>> On 09/08/2013 02:46 PM, Christoph Buchner wrote:
>>>
>>> On 09/07/2013 04:56 PM, Ned Batchelder wrote:
>>>> On 9/6/13 8:35 AM, Christoph Buchner wrote:
>>>>> Ned Batchelder schrieb am 06.09.2013 04:36:
>>>>>
>>>>>> On 9/5/13 6:33 AM, Christoph Buchner wrote:
>>>>>>> Sure, see attachment. step-by-step instructions in Instructions.txt
>>>>>>>
>>>>>>> thanks,
>>>>>>> Christoph
>>>>>> Christoph, when I follow your instructions, I get the same output for
>>>>>> both commands.  They both print "--Contents of arg_file.txt--", 
>>>>>> and both
>>>>>> say 0% coverage for my_script.py when the report is generated.  I've
>>>>>> attached the complete output.
>>>>>>
>>>>>> I'll look into why the script isn't measured (and why there's no 
>>>>>> debug
>>>>>> message that it isn't being measured), but it's disconcerting that 
>>>>>> our
>>>>>> results are different.
>>>>>>
>>>>>> --Ned.
>>>>>>
>>>>>>
>>>>> this is indeed weird! I'll try this again on my other machine, but 
>>>>> I'm pretty sure it showed the same behaviour.
>>>>>
>>>>> I haven't run my tests in a virtual environment, should I look into 
>>>>> setting one up, too, and try again?
>>>>>
>>>>> I'm already curious what you find out about why the script is not 
>>>>> measured, and no debug output about it either!
>>>>>
>>>>> best,
>>>>> Christoph
>>>>
>>>> I've learned many things...
>>>>
>>>> 1) The difference in behavior for our two configurations is the 
>>>> virtualenv: my_script.py has a shebang line of /usr/bin/python, 
>>>> which forced it out of my virtualenv, onto a Python that was not 
>>>> configured for coverage.  I changed it to "/usr/bin/env python", 
>>>> which is always a better choice.
>>>>
>>>> 2) The test_in_tmpdir case then wasn't getting debug tracing because 
>>>> the COVERAGE_PROCESS_START environment variable is set to 
>>>> ".coveragerc".  When your program is run in the tmp dir, that 
>>>> relative path name references a file that does not exist (in the tmp 
>>>> dir).  So you are using coverage.py defaults instead of your own 
>>>> settings.  This explains why I didn't even get debug tracing for the 
>>>> subprocess in the tmp dir.
>>>>
>>>> 3) Because your program runs in the tmp dir, the coverage data file 
>>>> is written to a data file in the tmp dir.  That file will not be 
>>>> combined or read later during the reporting phase.  In fact, it 
>>>> probably no longer exists, having been cleaned up with the rest of 
>>>> the tmp dir.
>>>>
>>>> To fix your problem:
>>>> a) export COVERAGE_PROCESS_START=/full/path/to/.coveragerc
>>>> b) In your .coveragerc:
>>>>         [run]
>>>>         data_file = /full/path/to/.coverage
>>>>
>>>> I hope you get the same (good) results I did.
>>>>
>>>> --Ned
>>>
>>> Ned,
>>> this is great news!
>>> re 1): Huh, the shebang. Makes sense, but I didn't think of that.
>>>
>>> re 2): This makes sense. I have observed something slightly 
>>> different, though:
>>> Even with the environment variable set to ".coveragerc", when running 
>>> the tmpdir test, I can control if I get debug output or not with the 
>>> .coveragerc in pytest-experiment, i.e. the debug option has an 
>>> effect! Apparently, pytest-experiment/.coveragerc still gets parsed 
>>> at some point, but not all options are effective (e.g. "parallel" I 
>>> think). I'm not sure if this something strange on my end, though.
>>>
>>> re 3): The tmp directories remain to exist (until next boot), you can 
>>> find them in /tmp/pytest-N, where N is a number incremented for every 
>>> test run. And yes, coverage files end up in there. Great find!
>>>
>>> If I follow both your fixes, I get correct coverage information for 
>>> the tmpdir test, so that's great! Thank you very much for finding this!
>>>
>>> Unfortunately, while the fix mechanism lets me continue working, it's 
>>> not a portable solution. The full-path environment variable has no 
>>> big impact except a bit bigger complexity (I still have to check if 
>>> pytest-cov, where you don't have to set the variable at all, uses a 
>>> full path). The full-path entry in .coveragerc is a bigger problem, 
>>> since I end up with a user-specific path in a program that I want to 
>>> share with others. So, everyone who wants to test coverage has to 
>>> modify .coveragerc to his own paths.
>>> I'm wondering if there's a more portable solution to this, e.g. by 
>>> saying "data_file = ./.coverage", since .coverage will be in the same 
>>> folder as .coveragerc - does coverage understand this?
>>>
>>>
>>> I have found some other strange things while continuing work:
>>>
>>> a) "coverage combine" apparently combines all the .coverage.name.X.Y 
>>> files, not only from the most recent run. This was surprising to me, 
>>> apparently you have to run "coverage combine", then "coverage erase" 
>>> to get rid of previous runs (see c)). I guess it makes sense for some 
>>> workflows, it was just surprising to me.
> "coverage combine" removes the files as it combines them.  You are 
> seeing extra files because you have the subprocess measurement in place, 
> which means any Python program you run will leave .coverage.* files 
> behind, including coverage.py!
>>>
>>> b) Apparently, when "parallel" is on (as it has to be for subprocess 
>>> coverage, right?), every coverage operation (even "coverage help" or 
>>> "coverage combine") creates an additional .coverage.name.X.Y file - 
>>> is that by design, or a bug?
> This is because COVERAGE_PROCESS_START is set.
>>>
>>> c) Coverage erase help says you use it to "Erase previously collected 
>>> coverage data.". However, it only erases .coverage files, not the 
>>> .coverage.name.X.Y files. So if you run, say, 5 tests without 
>>> combining, then say "coverage erase", you don't get a clean slate. If 
>>> you then run coverage another time, and then combine, the previous 
>>> coverage results get merged/combined with the current run, which 
>>> screws up coverage reports. My (fresh user) impression was that 
>>> coverage erase would erase all generated coverage files.
>>> This also clashes somehow with the --append option, since it 
>>> effectively "appends" on combine even if you don't specify that option.
> It does make sense for the "erase" command to remove all the data files 
> when parallel=true, I'm not sure why it's never been requested (or 
> implemented!) before...
>>>
>>> d) This is more of a feature request, but seeing as the command chain 
>>> of erase-run-combine-report seems to be pretty mandatory, it would be 
>>> great if coverage run would learn additional arguments to run other 
>>> commands this in one go, e.g. something like "--erase --combine 
>>> --report/--html"
> I've wondered about that, but I figure it's still just as easy to run 
> multiple coverage commands, and trying to combine them with switches 
> like that make it hard to control those commands.
>>>
>>> I don't have a bitbucket account, but if you want I can submit those 
>>> things as issues on the tracker.
>>>
>>> thanks for you help, it's very appreciated,
>>> Christoph
>>>
>> Ned,
>> in the meantime I've found that the limitation of having to specify 
>> full paths in .coveragerc also extends to other flags.
>> For example, I want to use "include" to only include my_script.py in 
>> the report to get rid of all the other files coverage also reports on, 
>> stuff in /usr/share/pyshared, 
>> /usr/local/lib/python2.7/dist-packages/py/ etc.. This only works if I 
>> use the full path to my_script.py in .coveragerc. I guess this is 
>> because .coveragerc is interpreted with respect to "current directory" 
>> (as it says in the docs) and not with respect to the _location_ of 
>> .coveragerc. I guess this is very often the same thing, but not in my 
>> case. Would it make more sense to switch to a "location" 
>> interpretation, or does that break other workflows?
> I can see that this is a useful case, I'd have to think carefully about 
> how to shift the interpretation, or make it switchable.  I definitely 
> want to make .coveragerc files useful in modern test scenarios.
> 
> --Ned.
>>
>> Also, pytest-cov apparently can't deal with my situation correctly, 
>> which doesn't surprise me very much. I guess I'll use pure coverage 
>> instead, although pytest-cov was very convenient to use (i.e. just add 
>> two flags to also get a coverage report when running py.test).
>>
>> best,
>> Christoph
>>
> 

a), b): Makes sense, I didn't think of that. Currently, I'm setting up a script for running coverage, combining, exporting, etc, so this shouldn't be a problem anymore as the env is local to the script, then.
c): I guess it was just an edge-case oversight? Good that you agree, in any case. :-)
d) you're probably right about combining with switches. I'll make a script running the commands in order.

cwd vs. file location: I agree, this will need careful thought. A switch/option for .coveragerc sounds like a good approach, for (potential) users who rely on the current directory interpretation.

best,
christoph