[TIP] Coverage does not cover script if py.test executes it from another directory

[Ned Batchelder]

On 9/8/13 10:23 AM, Christoph Buchner wrote:
>
> On 09/08/2013 02:46 PM, Christoph Buchner wrote:
>>
>> On 09/07/2013 04:56 PM, Ned Batchelder wrote:
>>> On 9/6/13 8:35 AM, Christoph Buchner wrote:
>>>> Ned Batchelder schrieb am 06.09.2013 04:36:
>>>>
>>>>> On 9/5/13 6:33 AM, Christoph Buchner wrote:
>>>>>> Sure, see attachment. step-by-step instructions in Instructions.txt
>>>>>>
>>>>>> thanks,
>>>>>> Christoph
>>>>> Christoph, when I follow your instructions, I get the same output for
>>>>> both commands.  They both print "--Contents of arg_file.txt--", 
>>>>> and both
>>>>> say 0% coverage for my_script.py when the report is generated.  I've
>>>>> attached the complete output.
>>>>>
>>>>> I'll look into why the script isn't measured (and why there's no 
>>>>> debug
>>>>> message that it isn't being measured), but it's disconcerting that 
>>>>> our
>>>>> results are different.
>>>>>
>>>>> --Ned.
>>>>>
>>>>>
>>>> this is indeed weird! I'll try this again on my other machine, but 
>>>> I'm pretty sure it showed the same behaviour.
>>>>
>>>> I haven't run my tests in a virtual environment, should I look into 
>>>> setting one up, too, and try again?
>>>>
>>>> I'm already curious what you find out about why the script is not 
>>>> measured, and no debug output about it either!
>>>>
>>>> best,
>>>> Christoph
>>>
>>> I've learned many things...
>>>
>>> 1) The difference in behavior for our two configurations is the 
>>> virtualenv: my_script.py has a shebang line of /usr/bin/python, 
>>> which forced it out of my virtualenv, onto a Python that was not 
>>> configured for coverage.  I changed it to "/usr/bin/env python", 
>>> which is always a better choice.
>>>
>>> 2) The test_in_tmpdir case then wasn't getting debug tracing because 
>>> the COVERAGE_PROCESS_START environment variable is set to 
>>> ".coveragerc".  When your program is run in the tmp dir, that 
>>> relative path name references a file that does not exist (in the tmp 
>>> dir).  So you are using coverage.py defaults instead of your own 
>>> settings.  This explains why I didn't even get debug tracing for the 
>>> subprocess in the tmp dir.
>>>
>>> 3) Because your program runs in the tmp dir, the coverage data file 
>>> is written to a data file in the tmp dir.  That file will not be 
>>> combined or read later during the reporting phase.  In fact, it 
>>> probably no longer exists, having been cleaned up with the rest of 
>>> the tmp dir.
>>>
>>> To fix your problem:
>>> a) export COVERAGE_PROCESS_START=/full/path/to/.coveragerc
>>> b) In your .coveragerc:
>>>         [run]
>>>         data_file = /full/path/to/.coverage
>>>
>>> I hope you get the same (good) results I did.
>>>
>>> --Ned
>>
>> Ned,
>> this is great news!
>> re 1): Huh, the shebang. Makes sense, but I didn't think of that.
>>
>> re 2): This makes sense. I have observed something slightly 
>> different, though:
>> Even with the environment variable set to ".coveragerc", when running 
>> the tmpdir test, I can control if I get debug output or not with the 
>> .coveragerc in pytest-experiment, i.e. the debug option has an 
>> effect! Apparently, pytest-experiment/.coveragerc still gets parsed 
>> at some point, but not all options are effective (e.g. "parallel" I 
>> think). I'm not sure if this something strange on my end, though.
>>
>> re 3): The tmp directories remain to exist (until next boot), you can 
>> find them in /tmp/pytest-N, where N is a number incremented for every 
>> test run. And yes, coverage files end up in there. Great find!
>>
>> If I follow both your fixes, I get correct coverage information for 
>> the tmpdir test, so that's great! Thank you very much for finding this!
>>
>> Unfortunately, while the fix mechanism lets me continue working, it's 
>> not a portable solution. The full-path environment variable has no 
>> big impact except a bit bigger complexity (I still have to check if 
>> pytest-cov, where you don't have to set the variable at all, uses a 
>> full path). The full-path entry in .coveragerc is a bigger problem, 
>> since I end up with a user-specific path in a program that I want to 
>> share with others. So, everyone who wants to test coverage has to 
>> modify .coveragerc to his own paths.
>> I'm wondering if there's a more portable solution to this, e.g. by 
>> saying "data_file = ./.coverage", since .coverage will be in the same 
>> folder as .coveragerc - does coverage understand this?
>>
>>
>> I have found some other strange things while continuing work:
>>
>> a) "coverage combine" apparently combines all the .coverage.name.X.Y 
>> files, not only from the most recent run. This was surprising to me, 
>> apparently you have to run "coverage combine", then "coverage erase" 
>> to get rid of previous runs (see c)). I guess it makes sense for some 
>> workflows, it was just surprising to me.
"coverage combine" removes the files as it combines them.  You are 
seeing extra files because you have the subprocess measurement in place, 
which means any Python program you run will leave .coverage.* files 
behind, including coverage.py!
>>
>> b) Apparently, when "parallel" is on (as it has to be for subprocess 
>> coverage, right?), every coverage operation (even "coverage help" or 
>> "coverage combine") creates an additional .coverage.name.X.Y file - 
>> is that by design, or a bug?
This is because COVERAGE_PROCESS_START is set.
>>
>> c) Coverage erase help says you use it to "Erase previously collected 
>> coverage data.". However, it only erases .coverage files, not the 
>> .coverage.name.X.Y files. So if you run, say, 5 tests without 
>> combining, then say "coverage erase", you don't get a clean slate. If 
>> you then run coverage another time, and then combine, the previous 
>> coverage results get merged/combined with the current run, which 
>> screws up coverage reports. My (fresh user) impression was that 
>> coverage erase would erase all generated coverage files.
>> This also clashes somehow with the --append option, since it 
>> effectively "appends" on combine even if you don't specify that option.
It does make sense for the "erase" command to remove all the data files 
when parallel=true, I'm not sure why it's never been requested (or 
implemented!) before...
>>
>> d) This is more of a feature request, but seeing as the command chain 
>> of erase-run-combine-report seems to be pretty mandatory, it would be 
>> great if coverage run would learn additional arguments to run other 
>> commands this in one go, e.g. something like "--erase --combine 
>> --report/--html"
I've wondered about that, but I figure it's still just as easy to run 
multiple coverage commands, and trying to combine them with switches 
like that make it hard to control those commands.
>>
>> I don't have a bitbucket account, but if you want I can submit those 
>> things as issues on the tracker.
>>
>> thanks for you help, it's very appreciated,
>> Christoph
>>
> Ned,
> in the meantime I've found that the limitation of having to specify 
> full paths in .coveragerc also extends to other flags.
> For example, I want to use "include" to only include my_script.py in 
> the report to get rid of all the other files coverage also reports on, 
> stuff in /usr/share/pyshared, 
> /usr/local/lib/python2.7/dist-packages/py/ etc.. This only works if I 
> use the full path to my_script.py in .coveragerc. I guess this is 
> because .coveragerc is interpreted with respect to "current directory" 
> (as it says in the docs) and not with respect to the _location_ of 
> .coveragerc. I guess this is very often the same thing, but not in my 
> case. Would it make more sense to switch to a "location" 
> interpretation, or does that break other workflows?
I can see that this is a useful case, I'd have to think carefully about 
how to shift the interpretation, or make it switchable.  I definitely 
want to make .coveragerc files useful in modern test scenarios.

--Ned.
>
> Also, pytest-cov apparently can't deal with my situation correctly, 
> which doesn't surprise me very much. I guess I'll use pure coverage 
> instead, although pytest-cov was very convenient to use (i.e. just add 
> two flags to also get a coverage report when running py.test).
>
> best,
> Christoph
>