[TIP] Coverage does not cover script if py.test executes it from another directory

Thu Sep 5 01:56:42 PDT 2013

Ned Batchelder schrieb am 05.09.2013 02:03:

> 
> On 9/4/13 5:45 PM, Christoph Buchner wrote:
>>
>> On 09/04/2013 05:03 PM, Christoph Buchner wrote:
>>> Ned Batchelder schrieb am 04.09.2013 16:11:
>>>
>>>> On 9/2/13 5:26 PM, Christoph Buchner wrote:
>>>>> On 09/02/2013 07:13 PM, Ned Batchelder wrote:
>>>>>> On 9/2/13 11:45 AM, Christoph Buchner wrote:
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> I have run into an issue using py.test, pytest-cov and coverage. The
>>>>>>> question is also on Stackoverflow in all detail
>>>>>>> (http://stackoverflow.com/q/18573542/599884), but I'll also
>>>>>>> summarize it here:
>>>>>>>
>>>>>>> I'm writing a python script which takes command line arguments,
>>>>>>> working with
>>>>>>> some files (git repos etc).
>>>>>>> I've written succeeding tests with py.test putting this script
>>>>>>> through its
>>>>>>> paces (verifying it works correctly with different arguments, etc),
>>>>>>> executing it with subprocess.call in the tests.
>>>>>>>
>>>>>>> Now I want to analyze code coverage with coverage.py.
>>>>>>> Coverage, when used via the pytest-cov plugin (which has
>>>>>>> subprocess-handling
>>>>>>> already built-in/ootb), does not see/cover my script when the 
>>>>>>> script is
>>>>>>> called from a temporary testing directory created with py.test's 
>>>>>>> tmpdir
>>>>>>> fixture.
>>>>>>> Coverage does see my script when it's called in the directory it
>>>>>>> resides in
>>>>>>> (and the filename argument points to a remote path).
>>>>>>>
>>>>>>> To illustrate with a minimal example,
>>>>>>>> Running in directory /home/cbuchner/pytest_experiment
>>>>>>>> Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
>>>>>>> gets correct coverage measurement, but
>>>>>>>> Running in directory /tmp/pytest-52/test_11
>>>>>>>> Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
>>>>>>> gives "Coverage.py warning: No data was collected."
>>>>>>> See http://stackoverflow.com/q/18573542/599884 for full
>>>>>>> console output and link to the used code.
>>>>>>>
>>>>>>> In both situations, my tests pass! This is on Ubuntu, with Coverage
>>>>>>> 3.6,
>>>>>>> pytest-2.3.5, pytest-cov 1.6, all from PyPi.
>>>>>>>
>>>>>>> Questions:
>>>>>>> * How can I get coverage to recognize my script even if it's
>>>>>>> executed in
>>>>>>> another directory?
>>>>>>> * Is this a bug in coverage, or something which is just not possible
>>>>>>> to do?
>>>>>>> Would be surprised if the latter, after all, tmpdir is a stock
>>>>>>> mechanism of
>>>>>>> py.test, right?
>>>>>>> * If the way I did it is an inappropriate testing implementation,
>>>>>>> I'd be
>>>>>>> grateful for pointers how to "properly" coverage-test 
>>>>>>> file-manipulating
>>>>>>> console utilities written in Python. Writing automated tests is
>>>>>>> pretty new
>>>>>>> to me, so I'm learning much and know little (yet).
>>>>>>>
>>>>>>> thanks for any pointers on this,
>>>>>>> best,
>>>>>>> Christoph
>>>>>>>
>>>>>> It's a little hard to know why it's not working without more details,
>>>>>> but a first thing that comes to mind is that coverage ignores code it
>>>>>> thinks is in the stdlib.  If your script is being run from a
>>>>>> directory under the stdlib somehow, then that could explain it.
>>>>>>
>>>>>> Can you run "coverage debug sys" in the same context as your failing
>>>>>> scenario?  It will show details of the configuration.
>>>>>>
>>>>>> --Ned.
>>>>> Yes, that's why I gave as much detail as possible (including code) in
>>>>> the SO answer. Didn't think of coverage debug sys, though, sorry. >.<
>>>>> I'm at another machine now, which exhibits the same problem.
>>>>> Find attached a console log (it's too long for an email I think). I
>>>>> don't think the paths I'm working in are under the stdlib. At the end
>>>>> of the log, I also ran the py.test with coverage with "-s", too, and
>>>>> get two additional warnings for the test_in_tmpdir.
>>>>> I can give a coverage debug sys output from the other, original,
>>>>> machine tomorrow, if still needed?
>>>>>
>>>>> Also, the code is at https://gist.github.com/bilderbuchi/6412754 (you
>>>>> can download a tar.gz of it), if you want to try and reproduce the
>>>>> problem.
>>>>>
>>>>> thanks a lot,
>>>>> Christoph
>>>> Christoph, instead of spending time reproducing your problem, I tried
>>>> adding a feature to coverage.py to help you diagnose it. Attached is a
>>>> tarball of the coverage.py kit.  It adds a flag:
>>>>
>>>>      coverage run --debug=notrace  myprog.py etc...
>>>>
>>>> This will make coverage.py write to stderr for every file it decides 
>>>> not
>>>> to trace.  Each file is listed with the reason it isn't being traced.
>>>> Can you try your scenario with it, and see if it gives you useful
>>>> information?
>>>>
>>>> --Ned.
>>>>
>>> Ned,
>>> thanks for the patched version! Unfortunately, I can't seem to 
>>> trigger that option from a .coveragerc containing
>>> [run]
>>> debug = notrace
>>> when doing $py.test --cov-config=.coveragerc --cov=my_script.py 
>>> tests/test_in_tmpdir.py
>>> (it works in coverage itself cause I see the output when using the 
>>> `coverage` command. Doesn't show anything useful though due to not 
>>> being modded for subprocess tracking yet)
>>> I have been using pytest-cov so far, which has subprocess covering 
>>> out-of-the-box. I am a bit hesitant fiddling with my (working) python 
>>> environment.
>>> Also, pytest-cov is proven to work(!) with the test_in_scriptdir 
>>> test, where I get coverage, so a general malfunction there can be 
>>> excluded.
>>>
>>> Reproducing my problem would be easy, though:
>>> * download the files, put the appropriate ones into a tests/ 
>>> directory as noted in the comments within (github gists sadly don't 
>>> offer subdirectories)
>>> * install pytest-cov if necessary
>>> * run the commands from the log file I attached to my previous mail
>>> * see if you get the same problem with the test_in_tmpdir.
>>>
>>> Anything further I can do to diagnose this?
>>> I can later try modifying my python install according to the guide to 
>>> make vanilla coverage work, to be able to use your patched version to 
>>> reproduce.
>>>
>>> thanks,
>>> Christoph
>>>
>>> _______________________________________________
>>> testing-in-python mailing list
>>> testing-in-python at lists.idyll.org
>>> http://lists.idyll.org/listinfo/testing-in-python
>> OK, after some further investigations on my home system, where I have 
>> enticed vanilla coverage to cover subprocesses (going the .pth route), 
>> with your patched version, it finds the script file for both tests, 
>> but doesn't cover it in case of test_tmpdir. See attached logfile 
>> console_log2.txt.
>>
>> best,
>> many thanks for any help,
>> christoph
> 
> Christoph, my best guess is that pytest-cov doesn't properly account for 
> the tmpdir feature of py.test.  When you provide --cov=my_script.py, 
> pytest-cov passes that to coverage.py as the "source" parameter.  
> Coverage.py will not measure code outside of "source", so when you run 
> the test in a tmpdir, the code being run has been copied outside of the 
> location coverage.py was told to focus on, and so coverage.py refuses to 
> measure it.
> 
> I don't understand why my_script.py doesn't appear in the debug=notrace 
> output, perhaps that setting isn't being used in the subprocess?
> 
> Can you try the tests again without the --cov switch, or is that the 
> only way to request coverage measurement?
> 
> --Ned.
> 

Ned, 

The concise summary: 
I think it's possible the core of the problem is that when my_script.py is called _from another directory_ via <path-to-script>/my_script.py, coverage for some reason (some path assumptions maybe?) does not pick up on covered lines, although it recognizes the existence of my_script.py itself correctly.

The detailed version:
There's several problems here that I see:
#1: pytest-cov doesn't seem to see my_script when I run the tmpdir test. It's difficult to debug why this happens, because of
#2: pytest-cov does not seem to correctly parse the debug=notrace option when given in a .coveragerc file, leading to no debug output (as seen at the end of console_log2.txt)

To exclude any influence and/or bugs within pytest-cov (so we don't get distracted by them, let's forget about pytest-cov for a while), I have run coverage itself (not via `py.test --cov`), as shown in console_log2.txt. 
The first test run with the script_dir test shows that subprocess coverage works.
The second test run with the tmpdir_test exposes
#3: coverage picks up my_script, but it shows 0% coverage (although the test runs the script correctly)!
So #3 is a bug in coverage I think, as there's no pytest-cov involved at all. Possibly #1 is triggered by #3 and would go away if #3 is fixed.

> Coverage.py will not measure code outside of "source", so when you run 
> the test in a tmpdir, the code being run has been copied outside of the 
> location coverage.py was told to focus on, and so coverage.py refuses to 
> measure it.

This is not correct. The only file being copied to tmpdir is arg_file.txt. The script stays in place. 
The only difference between the two tests is that once my_script.py called from within the script directory with
./my_script.py <path-to-tmp>/arg_file.txt (this works)
and in the other test it's called from within the tmpdir directory with
<path-to-script>/my_script.py arg_file.txt
and it's the latter test that doesn't show coverage on my_script.py.

> I don't understand why my_script.py doesn't appear in the debug=notrace 
> output, perhaps that setting isn't being used in the subprocess?

It doesn't appear in the pure coverage calls because it has been picked up, just shows 0 coverage (as evidenced by "my_script 3 3 0%" in L283 of console_log2.txt)
It doesn't appear in the py.test --cov calls because here the debug option does not seem to be picked up, as there's no debug output at all (see L291 ff.)

> Can you try the tests again without the --cov switch, or is that the 
> only way to request coverage measurement?
Without --cov, py.test only runs the test, no coverage reporting is done. But as I said, let's focus on pure coverage (i.e. #3), and leave pytest-cov out of this for now.

all the best,
Christoph