[TIP] Coverage does not cover script if py.test executes it from another directory

[Christoph Buchner]

Ned Batchelder schrieb am 05.09.2013 11:38:

> 
> On 9/5/13 4:56 AM, Christoph Buchner wrote:
>>
>>
>> Ned Batchelder schrieb am 05.09.2013 02:03:
>>
>>> On 9/4/13 5:45 PM, Christoph Buchner wrote:
>>>> On 09/04/2013 05:03 PM, Christoph Buchner wrote:
>>>>> Ned Batchelder schrieb am 04.09.2013 16:11:
>>>>>
>>>>>> On 9/2/13 5:26 PM, Christoph Buchner wrote:
>>>>>>> On 09/02/2013 07:13 PM, Ned Batchelder wrote:
>>>>>>>> On 9/2/13 11:45 AM, Christoph Buchner wrote:
>>>>>>>>> Hi folks,
>>>>>>>>>
>>>>>>>>> I have run into an issue using py.test, pytest-cov and coverage.
>>>>>>>>> The
>>>>>>>>> question is also on Stackoverflow in all detail
>>>>>>>>> (http://stackoverflow.com/q/18573542/599884), but I'll also
>>>>>>>>> summarize it here:
>>>>>>>>>
>>>>>>>>> I'm writing a python script which takes command line arguments,
>>>>>>>>> working with
>>>>>>>>> some files (git repos etc).
>>>>>>>>> I've written succeeding tests with py.test putting this script
>>>>>>>>> through its
>>>>>>>>> paces (verifying it works correctly with different arguments,
>>>>>>>>> etc),
>>>>>>>>> executing it with subprocess.call in the tests.
>>>>>>>>>
>>>>>>>>> Now I want to analyze code coverage with coverage.py.
>>>>>>>>> Coverage, when used via the pytest-cov plugin (which has
>>>>>>>>> subprocess-handling
>>>>>>>>> already built-in/ootb), does not see/cover my script when the
>>>>>>>>> script is
>>>>>>>>> called from a temporary testing directory created with py.test's
>>>>>>>>> tmpdir
>>>>>>>>> fixture.
>>>>>>>>> Coverage does see my script when it's called in the directory it
>>>>>>>>> resides in
>>>>>>>>> (and the filename argument points to a remote path).
>>>>>>>>>
>>>>>>>>> To illustrate with a minimal example,
>>>>>>>>>> Running in directory /home/cbuchner/pytest_experiment
>>>>>>>>>> Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
>>>>>>>>> gets correct coverage measurement, but
>>>>>>>>>> Running in directory /tmp/pytest-52/test_11
>>>>>>>>>> Command: /home/cbuchner/pytest_experiment/my_script.py
>>>>>>>>>> arg_file.txt
>>>>>>>>> gives "Coverage.py warning: No data was collected."
>>>>>>>>> See http://stackoverflow.com/q/18573542/599884 for full
>>>>>>>>> console output and link to the used code.
>>>>>>>>>
>>>>>>>>> In both situations, my tests pass! This is on Ubuntu, with
>>>>>>>>> Coverage
>>>>>>>>> 3.6,
>>>>>>>>> pytest-2.3.5, pytest-cov 1.6, all from PyPi.
>>>>>>>>>
>>>>>>>>> Questions:
>>>>>>>>> * How can I get coverage to recognize my script even if it's
>>>>>>>>> executed in
>>>>>>>>> another directory?
>>>>>>>>> * Is this a bug in coverage, or something which is just not
>>>>>>>>> possible
>>>>>>>>> to do?
>>>>>>>>> Would be surprised if the latter, after all, tmpdir is a stock
>>>>>>>>> mechanism of
>>>>>>>>> py.test, right?
>>>>>>>>> * If the way I did it is an inappropriate testing implementation,
>>>>>>>>> I'd be
>>>>>>>>> grateful for pointers how to "properly" coverage-test
>>>>>>>>> file-manipulating
>>>>>>>>> console utilities written in Python. Writing automated tests is
>>>>>>>>> pretty new
>>>>>>>>> to me, so I'm learning much and know little (yet).
>>>>>>>>>
>>>>>>>>> thanks for any pointers on this,
>>>>>>>>> best,
>>>>>>>>> Christoph
>>>>>>>>>
>>>>>>>> It's a little hard to know why it's not working without more
>>>>>>>> details,
>>>>>>>> but a first thing that comes to mind is that coverage ignores code
>>>>>>>> it
>>>>>>>> thinks is in the stdlib.  If your script is being run from a
>>>>>>>> directory under the stdlib somehow, then that could explain it.
>>>>>>>>
>>>>>>>> Can you run "coverage debug sys" in the same context as your
>>>>>>>> failing
>>>>>>>> scenario?  It will show details of the configuration.
>>>>>>>>
>>>>>>>> --Ned.
>>>>>>> Yes, that's why I gave as much detail as possible (including code)
>>>>>>> in
>>>>>>> the SO answer. Didn't think of coverage debug sys, though, sorry.
>>>>>>> >.<
>>>>>>> I'm at another machine now, which exhibits the same problem.
>>>>>>> Find attached a console log (it's too long for an email I think). I
>>>>>>> don't think the paths I'm working in are under the stdlib. At the
>>>>>>> end
>>>>>>> of the log, I also ran the py.test with coverage with "-s", too, and
>>>>>>> get two additional warnings for the test_in_tmpdir.
>>>>>>> I can give a coverage debug sys output from the other, original,
>>>>>>> machine tomorrow, if still needed?
>>>>>>>
>>>>>>> Also, the code is at https://gist.github.com/bilderbuchi/6412754
>>>>>>> (you
>>>>>>> can download a tar.gz of it), if you want to try and reproduce the
>>>>>>> problem.
>>>>>>>
>>>>>>> thanks a lot,
>>>>>>> Christoph
>>>>>> Christoph, instead of spending time reproducing your problem, I tried
>>>>>> adding a feature to coverage.py to help you diagnose it. Attached is
>>>>>> a
>>>>>> tarball of the coverage.py kit.  It adds a flag:
>>>>>>
>>>>>>       coverage run --debug=notrace  myprog.py etc...
>>>>>>
>>>>>> This will make coverage.py write to stderr for every file it decides
>>>>>> not
>>>>>> to trace.  Each file is listed with the reason it isn't being traced.
>>>>>> Can you try your scenario with it, and see if it gives you useful
>>>>>> information?
>>>>>>
>>>>>> --Ned.
>>>>>>
>>>>> Ned,
>>>>> thanks for the patched version! Unfortunately, I can't seem to
>>>>> trigger that option from a .coveragerc containing
>>>>> [run]
>>>>> debug = notrace
>>>>> when doing $py.test --cov-config=.coveragerc --cov=my_script.py
>>>>> tests/test_in_tmpdir.py
>>>>> (it works in coverage itself cause I see the output when using the
>>>>> `coverage` command. Doesn't show anything useful though due to not
>>>>> being modded for subprocess tracking yet)
>>>>> I have been using pytest-cov so far, which has subprocess covering
>>>>> out-of-the-box. I am a bit hesitant fiddling with my (working) python
>>>>> environment.
>>>>> Also, pytest-cov is proven to work(!) with the test_in_scriptdir
>>>>> test, where I get coverage, so a general malfunction there can be
>>>>> excluded.
>>>>>
>>>>> Reproducing my problem would be easy, though:
>>>>> * download the files, put the appropriate ones into a tests/
>>>>> directory as noted in the comments within (github gists sadly don't
>>>>> offer subdirectories)
>>>>> * install pytest-cov if necessary
>>>>> * run the commands from the log file I attached to my previous mail
>>>>> * see if you get the same problem with the test_in_tmpdir.
>>>>>
>>>>> Anything further I can do to diagnose this?
>>>>> I can later try modifying my python install according to the guide to
>>>>> make vanilla coverage work, to be able to use your patched version to
>>>>> reproduce.
>>>>>
>>>>> thanks,
>>>>> Christoph
>>>>>
>>>>> _______________________________________________
>>>>> testing-in-python mailing list
>>>>> testing-in-python at lists.idyll.org
>>>>> http://lists.idyll.org/listinfo/testing-in-python
>>>> OK, after some further investigations on my home system, where I have
>>>> enticed vanilla coverage to cover subprocesses (going the .pth route),
>>>> with your patched version, it finds the script file for both tests,
>>>> but doesn't cover it in case of test_tmpdir. See attached logfile
>>>> console_log2.txt.
>>>>
>>>> best,
>>>> many thanks for any help,
>>>> christoph
>>> Christoph, my best guess is that pytest-cov doesn't properly account for
>>> the tmpdir feature of py.test.  When you provide --cov=my_script.py,
>>> pytest-cov passes that to coverage.py as the "source" parameter.
>>> Coverage.py will not measure code outside of "source", so when you run
>>> the test in a tmpdir, the code being run has been copied outside of the
>>> location coverage.py was told to focus on, and so coverage.py refuses to
>>> measure it.
>>>
>>> I don't understand why my_script.py doesn't appear in the debug=notrace
>>> output, perhaps that setting isn't being used in the subprocess?
>>>
>>> Can you try the tests again without the --cov switch, or is that the
>>> only way to request coverage measurement?
>>>
>>> --Ned.
>>>
>> Ned,
>>
>> The concise summary:
>> I think it's possible the core of the problem is that when my_script.py
>> is called _from another directory_ via <path-to-script>/my_script.py,
>> coverage for some reason (some path assumptions maybe?) does not pick up
>> on covered lines, although it recognizes the existence of my_script.py
>> itself correctly.
>>
>> The detailed version:
>> There's several problems here that I see:
>> #1: pytest-cov doesn't seem to see my_script when I run the tmpdir test.
>> It's difficult to debug why this happens, because of
>> #2: pytest-cov does not seem to correctly parse the debug=notrace option
>> when given in a .coveragerc file, leading to no debug output (as seen at
>> the end of console_log2.txt)
>>
>> To exclude any influence and/or bugs within pytest-cov (so we don't get
>> distracted by them, let's forget about pytest-cov for a while), I have
>> run coverage itself (not via `py.test --cov`), as shown in
>> console_log2.txt.
>> The first test run with the script_dir test shows that subprocess
>> coverage works.
>> The second test run with the tmpdir_test exposes
>> #3: coverage picks up my_script, but it shows 0% coverage (although the
>> test runs the script correctly)!
>> So #3 is a bug in coverage I think, as there's no pytest-cov involved at
>> all. Possibly #1 is triggered by #3 and would go away if #3 is fixed.
>>
>>> Coverage.py will not measure code outside of "source", so when you run
>>> the test in a tmpdir, the code being run has been copied outside of the
>>> location coverage.py was told to focus on, and so coverage.py refuses to
>>> measure it.
>> This is not correct. The only file being copied to tmpdir is
>> arg_file.txt. The script stays in place.
>> The only difference between the two tests is that once my_script.py
>> called from within the script directory with
>> ./my_script.py <path-to-tmp>/arg_file.txt (this works)
>> and in the other test it's called from within the tmpdir directory with
>> <path-to-script>/my_script.py arg_file.txt
>> and it's the latter test that doesn't show coverage on my_script.py.
>>
>>> I don't understand why my_script.py doesn't appear in the debug=notrace
>>> output, perhaps that setting isn't being used in the subprocess?
>> It doesn't appear in the pure coverage calls because it has been picked
>> up, just shows 0 coverage (as evidenced by "my_script 3 3 0%" in L283 of
>> console_log2.txt)
>> It doesn't appear in the py.test --cov calls because here the debug
>> option does not seem to be picked up, as there's no debug output at all
>> (see L291 ff.)
>>
>>> Can you try the tests again without the --cov switch, or is that the
>>> only way to request coverage measurement?
>> Without --cov, py.test only runs the test, no coverage reporting is done.
>> But as I said, let's focus on pure coverage (i.e. #3), and leave
>> pytest-cov out of this for now.
>>
>> all the best,
>> Christoph
> 
> OK, can you provide me with detailed instructions and ideally a tarball 
> of files that I can use to reproduce the problem?
> 
> --Ned.
> 

Sure, see attachment. step-by-step instructions in Instructions.txt

thanks,
Christoph
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pytest_experiment_for_Ned.tar.gz
Type: application/x-gzip
Size: 2067 bytes
Desc: not available
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20130905/897d8dd4/attachment.bin>