[TIP] Coverage does not cover script if py.test executes it from another directory
bilderbuchi at phononoia.at
Sun Sep 8 07:23:09 PDT 2013
On 09/08/2013 02:46 PM, Christoph Buchner wrote:
> On 09/07/2013 04:56 PM, Ned Batchelder wrote:
>> On 9/6/13 8:35 AM, Christoph Buchner wrote:
>>> Ned Batchelder schrieb am 06.09.2013 04:36:
>>>> On 9/5/13 6:33 AM, Christoph Buchner wrote:
>>>>> Sure, see attachment. step-by-step instructions in Instructions.txt
>>>> Christoph, when I follow your instructions, I get the same output for
>>>> both commands. They both print "--Contents of arg_file.txt--", and
>>>> say 0% coverage for my_script.py when the report is generated. I've
>>>> attached the complete output.
>>>> I'll look into why the script isn't measured (and why there's no debug
>>>> message that it isn't being measured), but it's disconcerting that our
>>>> results are different.
>>> this is indeed weird! I'll try this again on my other machine, but
>>> I'm pretty sure it showed the same behaviour.
>>> I haven't run my tests in a virtual environment, should I look into
>>> setting one up, too, and try again?
>>> I'm already curious what you find out about why the script is not
>>> measured, and no debug output about it either!
>> I've learned many things...
>> 1) The difference in behavior for our two configurations is the
>> virtualenv: my_script.py has a shebang line of /usr/bin/python, which
>> forced it out of my virtualenv, onto a Python that was not configured
>> for coverage. I changed it to "/usr/bin/env python", which is always
>> a better choice.
>> 2) The test_in_tmpdir case then wasn't getting debug tracing because
>> the COVERAGE_PROCESS_START environment variable is set to
>> ".coveragerc". When your program is run in the tmp dir, that
>> relative path name references a file that does not exist (in the tmp
>> dir). So you are using coverage.py defaults instead of your own
>> settings. This explains why I didn't even get debug tracing for the
>> subprocess in the tmp dir.
>> 3) Because your program runs in the tmp dir, the coverage data file
>> is written to a data file in the tmp dir. That file will not be
>> combined or read later during the reporting phase. In fact, it
>> probably no longer exists, having been cleaned up with the rest of
>> the tmp dir.
>> To fix your problem:
>> a) export COVERAGE_PROCESS_START=/full/path/to/.coveragerc
>> b) In your .coveragerc:
>> data_file = /full/path/to/.coverage
>> I hope you get the same (good) results I did.
> this is great news!
> re 1): Huh, the shebang. Makes sense, but I didn't think of that.
> re 2): This makes sense. I have observed something slightly different,
> Even with the environment variable set to ".coveragerc", when running
> the tmpdir test, I can control if I get debug output or not with the
> .coveragerc in pytest-experiment, i.e. the debug option has an effect!
> Apparently, pytest-experiment/.coveragerc still gets parsed at some
> point, but not all options are effective (e.g. "parallel" I think).
> I'm not sure if this something strange on my end, though.
> re 3): The tmp directories remain to exist (until next boot), you can
> find them in /tmp/pytest-N, where N is a number incremented for every
> test run. And yes, coverage files end up in there. Great find!
> If I follow both your fixes, I get correct coverage information for
> the tmpdir test, so that's great! Thank you very much for finding this!
> Unfortunately, while the fix mechanism lets me continue working, it's
> not a portable solution. The full-path environment variable has no big
> impact except a bit bigger complexity (I still have to check if
> pytest-cov, where you don't have to set the variable at all, uses a
> full path). The full-path entry in .coveragerc is a bigger problem,
> since I end up with a user-specific path in a program that I want to
> share with others. So, everyone who wants to test coverage has to
> modify .coveragerc to his own paths.
> I'm wondering if there's a more portable solution to this, e.g. by
> saying "data_file = ./.coverage", since .coverage will be in the same
> folder as .coveragerc - does coverage understand this?
> I have found some other strange things while continuing work:
> a) "coverage combine" apparently combines all the .coverage.name.X.Y
> files, not only from the most recent run. This was surprising to me,
> apparently you have to run "coverage combine", then "coverage erase"
> to get rid of previous runs (see c)). I guess it makes sense for some
> workflows, it was just surprising to me.
> b) Apparently, when "parallel" is on (as it has to be for subprocess
> coverage, right?), every coverage operation (even "coverage help" or
> "coverage combine") creates an additional .coverage.name.X.Y file - is
> that by design, or a bug?
> c) Coverage erase help says you use it to "Erase previously collected
> coverage data.". However, it only erases .coverage files, not the
> .coverage.name.X.Y files. So if you run, say, 5 tests without
> combining, then say "coverage erase", you don't get a clean slate. If
> you then run coverage another time, and then combine, the previous
> coverage results get merged/combined with the current run, which
> screws up coverage reports. My (fresh user) impression was that
> coverage erase would erase all generated coverage files.
> This also clashes somehow with the --append option, since it
> effectively "appends" on combine even if you don't specify that option.
> d) This is more of a feature request, but seeing as the command chain
> of erase-run-combine-report seems to be pretty mandatory, it would be
> great if coverage run would learn additional arguments to run other
> commands this in one go, e.g. something like "--erase --combine
> I don't have a bitbucket account, but if you want I can submit those
> things as issues on the tracker.
> thanks for you help, it's very appreciated,
in the meantime I've found that the limitation of having to specify full
paths in .coveragerc also extends to other flags.
For example, I want to use "include" to only include my_script.py in the
report to get rid of all the other files coverage also reports on, stuff
in /usr/share/pyshared, /usr/local/lib/python2.7/dist-packages/py/ etc..
This only works if I use the full path to my_script.py in .coveragerc. I
guess this is because .coveragerc is interpreted with respect to
"current directory" (as it says in the docs) and not with respect to the
_location_ of .coveragerc. I guess this is very often the same thing,
but not in my case. Would it make more sense to switch to a "location"
interpretation, or does that break other workflows?
Also, pytest-cov apparently can't deal with my situation correctly,
which doesn't surprise me very much. I guess I'll use pure coverage
instead, although pytest-cov was very convenient to use (i.e. just add
two flags to also get a coverage report when running py.test).
More information about the testing-in-python