[TIP] Coverage does not cover script if py.test executes it from another directory

[Christoph Buchner]

On 09/07/2013 04:56 PM, Ned Batchelder wrote:
> On 9/6/13 8:35 AM, Christoph Buchner wrote:
>> Ned Batchelder schrieb am 06.09.2013 04:36:
>>
>>> On 9/5/13 6:33 AM, Christoph Buchner wrote:
>>>> Sure, see attachment. step-by-step instructions in Instructions.txt
>>>>
>>>> thanks,
>>>> Christoph
>>> Christoph, when I follow your instructions, I get the same output for
>>> both commands.  They both print "--Contents of arg_file.txt--", and 
>>> both
>>> say 0% coverage for my_script.py when the report is generated.  I've
>>> attached the complete output.
>>>
>>> I'll look into why the script isn't measured (and why there's no debug
>>> message that it isn't being measured), but it's disconcerting that our
>>> results are different.
>>>
>>> --Ned.
>>>
>>>
>> this is indeed weird! I'll try this again on my other machine, but 
>> I'm pretty sure it showed the same behaviour.
>>
>> I haven't run my tests in a virtual environment, should I look into 
>> setting one up, too, and try again?
>>
>> I'm already curious what you find out about why the script is not 
>> measured, and no debug output about it either!
>>
>> best,
>> Christoph
>
> I've learned many things...
>
> 1) The difference in behavior for our two configurations is the 
> virtualenv: my_script.py has a shebang line of /usr/bin/python, which 
> forced it out of my virtualenv, onto a Python that was not configured 
> for coverage.  I changed it to "/usr/bin/env python", which is always 
> a better choice.
>
> 2) The test_in_tmpdir case then wasn't getting debug tracing because 
> the COVERAGE_PROCESS_START environment variable is set to 
> ".coveragerc".  When your program is run in the tmp dir, that relative 
> path name references a file that does not exist (in the tmp dir).  So 
> you are using coverage.py defaults instead of your own settings.  This 
> explains why I didn't even get debug tracing for the subprocess in the 
> tmp dir.
>
> 3) Because your program runs in the tmp dir, the coverage data file is 
> written to a data file in the tmp dir.  That file will not be combined 
> or read later during the reporting phase.  In fact, it probably no 
> longer exists, having been cleaned up with the rest of the tmp dir.
>
> To fix your problem:
> a) export COVERAGE_PROCESS_START=/full/path/to/.coveragerc
> b) In your .coveragerc:
>         [run]
>         data_file = /full/path/to/.coverage
>
> I hope you get the same (good) results I did.
>
> --Ned

Ned,
this is great news!
re 1): Huh, the shebang. Makes sense, but I didn't think of that.

re 2): This makes sense. I have observed something slightly different, 
though:
Even with the environment variable set to ".coveragerc", when running 
the tmpdir test, I can control if I get debug output or not with the 
.coveragerc in pytest-experiment, i.e. the debug option has an effect! 
Apparently, pytest-experiment/.coveragerc still gets parsed at some 
point, but not all options are effective (e.g. "parallel" I think). I'm 
not sure if this something strange on my end, though.

re 3): The tmp directories remain to exist (until next boot), you can 
find them in /tmp/pytest-N, where N is a number incremented for every 
test run. And yes, coverage files end up in there. Great find!

If I follow both your fixes, I get correct coverage information for the 
tmpdir test, so that's great! Thank you very much for finding this!

Unfortunately, while the fix mechanism lets me continue working, it's 
not a portable solution. The full-path environment variable has no big 
impact except a bit bigger complexity (I still have to check if 
pytest-cov, where you don't have to set the variable at all, uses a full 
path). The full-path entry in .coveragerc is a bigger problem, since I 
end up with a user-specific path in a program that I want to share with 
others. So, everyone who wants to test coverage has to modify 
.coveragerc to his own paths.
I'm wondering if there's a more portable solution to this, e.g. by 
saying "data_file = ./.coverage", since .coverage will be in the same 
folder as .coveragerc - does coverage understand this?


I have found some other strange things while continuing work:

a) "coverage combine" apparently combines all the .coverage.name.X.Y 
files, not only from the most recent run. This was surprising to me, 
apparently you have to run "coverage combine", then "coverage erase" to 
get rid of previous runs (see c)). I guess it makes sense for some 
workflows, it was just surprising to me.

b) Apparently, when "parallel" is on (as it has to be for subprocess 
coverage, right?), every coverage operation (even "coverage help" or 
"coverage combine") creates an additional .coverage.name.X.Y file - is 
that by design, or a bug?

c) Coverage erase help says you use it to "Erase previously collected 
coverage data.". However, it only erases .coverage files, not the 
.coverage.name.X.Y files. So if you run, say, 5 tests without combining, 
then say "coverage erase", you don't get a clean slate. If you then run 
coverage another time, and then combine, the previous coverage results 
get merged/combined with the current run, which screws up coverage 
reports. My (fresh user) impression was that coverage erase would erase 
all generated coverage files.
This also clashes somehow with the --append option, since it effectively 
"appends" on combine even if you don't specify that option.

d) This is more of a feature request, but seeing as the command chain of 
erase-run-combine-report seems to be pretty mandatory, it would be great 
if coverage run would learn additional arguments to run other commands 
this in one go, e.g. something like "--erase --combine --report/--html"

I don't have a bitbucket account, but if you want I can submit those 
things as issues on the tracker.

thanks for you help, it's very appreciated,
Christoph