[TIP] Coverage does not cover script if py.test executes it from another directory

Christoph Buchner bilderbuchi at phononoia.at
Sun Sep 8 07:23:09 PDT 2013

On 09/08/2013 02:46 PM, Christoph Buchner wrote:
> On 09/07/2013 04:56 PM, Ned Batchelder wrote:
>> On 9/6/13 8:35 AM, Christoph Buchner wrote:
>>> Ned Batchelder schrieb am 06.09.2013 04:36:
>>>> On 9/5/13 6:33 AM, Christoph Buchner wrote:
>>>>> Sure, see attachment. step-by-step instructions in Instructions.txt
>>>>> thanks,
>>>>> Christoph
>>>> Christoph, when I follow your instructions, I get the same output for
>>>> both commands.  They both print "--Contents of arg_file.txt--", and 
>>>> both
>>>> say 0% coverage for my_script.py when the report is generated.  I've
>>>> attached the complete output.
>>>> I'll look into why the script isn't measured (and why there's no debug
>>>> message that it isn't being measured), but it's disconcerting that our
>>>> results are different.
>>>> --Ned.
>>> this is indeed weird! I'll try this again on my other machine, but 
>>> I'm pretty sure it showed the same behaviour.
>>> I haven't run my tests in a virtual environment, should I look into 
>>> setting one up, too, and try again?
>>> I'm already curious what you find out about why the script is not 
>>> measured, and no debug output about it either!
>>> best,
>>> Christoph
>> I've learned many things...
>> 1) The difference in behavior for our two configurations is the 
>> virtualenv: my_script.py has a shebang line of /usr/bin/python, which 
>> forced it out of my virtualenv, onto a Python that was not configured 
>> for coverage.  I changed it to "/usr/bin/env python", which is always 
>> a better choice.
>> 2) The test_in_tmpdir case then wasn't getting debug tracing because 
>> the COVERAGE_PROCESS_START environment variable is set to 
>> ".coveragerc".  When your program is run in the tmp dir, that 
>> relative path name references a file that does not exist (in the tmp 
>> dir).  So you are using coverage.py defaults instead of your own 
>> settings.  This explains why I didn't even get debug tracing for the 
>> subprocess in the tmp dir.
>> 3) Because your program runs in the tmp dir, the coverage data file 
>> is written to a data file in the tmp dir.  That file will not be 
>> combined or read later during the reporting phase.  In fact, it 
>> probably no longer exists, having been cleaned up with the rest of 
>> the tmp dir.
>> To fix your problem:
>> a) export COVERAGE_PROCESS_START=/full/path/to/.coveragerc
>> b) In your .coveragerc:
>>         [run]
>>         data_file = /full/path/to/.coverage
>> I hope you get the same (good) results I did.
>> --Ned
> Ned,
> this is great news!
> re 1): Huh, the shebang. Makes sense, but I didn't think of that.
> re 2): This makes sense. I have observed something slightly different, 
> though:
> Even with the environment variable set to ".coveragerc", when running 
> the tmpdir test, I can control if I get debug output or not with the 
> .coveragerc in pytest-experiment, i.e. the debug option has an effect! 
> Apparently, pytest-experiment/.coveragerc still gets parsed at some 
> point, but not all options are effective (e.g. "parallel" I think). 
> I'm not sure if this something strange on my end, though.
> re 3): The tmp directories remain to exist (until next boot), you can 
> find them in /tmp/pytest-N, where N is a number incremented for every 
> test run. And yes, coverage files end up in there. Great find!
> If I follow both your fixes, I get correct coverage information for 
> the tmpdir test, so that's great! Thank you very much for finding this!
> Unfortunately, while the fix mechanism lets me continue working, it's 
> not a portable solution. The full-path environment variable has no big 
> impact except a bit bigger complexity (I still have to check if 
> pytest-cov, where you don't have to set the variable at all, uses a 
> full path). The full-path entry in .coveragerc is a bigger problem, 
> since I end up with a user-specific path in a program that I want to 
> share with others. So, everyone who wants to test coverage has to 
> modify .coveragerc to his own paths.
> I'm wondering if there's a more portable solution to this, e.g. by 
> saying "data_file = ./.coverage", since .coverage will be in the same 
> folder as .coveragerc - does coverage understand this?
> I have found some other strange things while continuing work:
> a) "coverage combine" apparently combines all the .coverage.name.X.Y 
> files, not only from the most recent run. This was surprising to me, 
> apparently you have to run "coverage combine", then "coverage erase" 
> to get rid of previous runs (see c)). I guess it makes sense for some 
> workflows, it was just surprising to me.
> b) Apparently, when "parallel" is on (as it has to be for subprocess 
> coverage, right?), every coverage operation (even "coverage help" or 
> "coverage combine") creates an additional .coverage.name.X.Y file - is 
> that by design, or a bug?
> c) Coverage erase help says you use it to "Erase previously collected 
> coverage data.". However, it only erases .coverage files, not the 
> .coverage.name.X.Y files. So if you run, say, 5 tests without 
> combining, then say "coverage erase", you don't get a clean slate. If 
> you then run coverage another time, and then combine, the previous 
> coverage results get merged/combined with the current run, which 
> screws up coverage reports. My (fresh user) impression was that 
> coverage erase would erase all generated coverage files.
> This also clashes somehow with the --append option, since it 
> effectively "appends" on combine even if you don't specify that option.
> d) This is more of a feature request, but seeing as the command chain 
> of erase-run-combine-report seems to be pretty mandatory, it would be 
> great if coverage run would learn additional arguments to run other 
> commands this in one go, e.g. something like "--erase --combine 
> --report/--html"
> I don't have a bitbucket account, but if you want I can submit those 
> things as issues on the tracker.
> thanks for you help, it's very appreciated,
> Christoph
in the meantime I've found that the limitation of having to specify full 
paths in .coveragerc also extends to other flags.
For example, I want to use "include" to only include my_script.py in the 
report to get rid of all the other files coverage also reports on, stuff 
in /usr/share/pyshared, /usr/local/lib/python2.7/dist-packages/py/ etc.. 
This only works if I use the full path to my_script.py in .coveragerc. I 
guess this is because .coveragerc is interpreted with respect to 
"current directory" (as it says in the docs) and not with respect to the 
_location_ of .coveragerc. I guess this is very often the same thing, 
but not in my case. Would it make more sense to switch to a "location" 
interpretation, or does that break other workflows?

Also, pytest-cov apparently can't deal with my situation correctly, 
which doesn't surprise me very much. I guess I'll use pure coverage 
instead, although pytest-cov was very convenient to use (i.e. just add 
two flags to also get a coverage report when running py.test).


More information about the testing-in-python mailing list