[TIP] Coverage does not cover script if py.test executes it from another directory

[Ned Batchelder]

On 9/5/13 4:56 AM, Christoph Buchner wrote:
>
>
> Ned Batchelder schrieb am 05.09.2013 02:03:
>
>> On 9/4/13 5:45 PM, Christoph Buchner wrote:
>>> On 09/04/2013 05:03 PM, Christoph Buchner wrote:
>>>> Ned Batchelder schrieb am 04.09.2013 16:11:
>>>>
>>>>> On 9/2/13 5:26 PM, Christoph Buchner wrote:
>>>>>> On 09/02/2013 07:13 PM, Ned Batchelder wrote:
>>>>>>> On 9/2/13 11:45 AM, Christoph Buchner wrote:
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> I have run into an issue using py.test, pytest-cov and coverage. The
>>>>>>>> question is also on Stackoverflow in all detail
>>>>>>>> (http://stackoverflow.com/q/18573542/599884), but I'll also
>>>>>>>> summarize it here:
>>>>>>>>
>>>>>>>> I'm writing a python script which takes command line arguments,
>>>>>>>> working with
>>>>>>>> some files (git repos etc).
>>>>>>>> I've written succeeding tests with py.test putting this script
>>>>>>>> through its
>>>>>>>> paces (verifying it works correctly with different arguments, etc),
>>>>>>>> executing it with subprocess.call in the tests.
>>>>>>>>
>>>>>>>> Now I want to analyze code coverage with coverage.py.
>>>>>>>> Coverage, when used via the pytest-cov plugin (which has
>>>>>>>> subprocess-handling
>>>>>>>> already built-in/ootb), does not see/cover my script when the
>>>>>>>> script is
>>>>>>>> called from a temporary testing directory created with py.test's
>>>>>>>> tmpdir
>>>>>>>> fixture.
>>>>>>>> Coverage does see my script when it's called in the directory it
>>>>>>>> resides in
>>>>>>>> (and the filename argument points to a remote path).
>>>>>>>>
>>>>>>>> To illustrate with a minimal example,
>>>>>>>>> Running in directory /home/cbuchner/pytest_experiment
>>>>>>>>> Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
>>>>>>>> gets correct coverage measurement, but
>>>>>>>>> Running in directory /tmp/pytest-52/test_11
>>>>>>>>> Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
>>>>>>>> gives "Coverage.py warning: No data was collected."
>>>>>>>> See http://stackoverflow.com/q/18573542/599884 for full
>>>>>>>> console output and link to the used code.
>>>>>>>>
>>>>>>>> In both situations, my tests pass! This is on Ubuntu, with Coverage
>>>>>>>> 3.6,
>>>>>>>> pytest-2.3.5, pytest-cov 1.6, all from PyPi.
>>>>>>>>
>>>>>>>> Questions:
>>>>>>>> * How can I get coverage to recognize my script even if it's
>>>>>>>> executed in
>>>>>>>> another directory?
>>>>>>>> * Is this a bug in coverage, or something which is just not possible
>>>>>>>> to do?
>>>>>>>> Would be surprised if the latter, after all, tmpdir is a stock
>>>>>>>> mechanism of
>>>>>>>> py.test, right?
>>>>>>>> * If the way I did it is an inappropriate testing implementation,
>>>>>>>> I'd be
>>>>>>>> grateful for pointers how to "properly" coverage-test
>>>>>>>> file-manipulating
>>>>>>>> console utilities written in Python. Writing automated tests is
>>>>>>>> pretty new
>>>>>>>> to me, so I'm learning much and know little (yet).
>>>>>>>>
>>>>>>>> thanks for any pointers on this,
>>>>>>>> best,
>>>>>>>> Christoph
>>>>>>>>
>>>>>>> It's a little hard to know why it's not working without more details,
>>>>>>> but a first thing that comes to mind is that coverage ignores code it
>>>>>>> thinks is in the stdlib.  If your script is being run from a
>>>>>>> directory under the stdlib somehow, then that could explain it.
>>>>>>>
>>>>>>> Can you run "coverage debug sys" in the same context as your failing
>>>>>>> scenario?  It will show details of the configuration.
>>>>>>>
>>>>>>> --Ned.
>>>>>> Yes, that's why I gave as much detail as possible (including code) in
>>>>>> the SO answer. Didn't think of coverage debug sys, though, sorry. >.<
>>>>>> I'm at another machine now, which exhibits the same problem.
>>>>>> Find attached a console log (it's too long for an email I think). I
>>>>>> don't think the paths I'm working in are under the stdlib. At the end
>>>>>> of the log, I also ran the py.test with coverage with "-s", too, and
>>>>>> get two additional warnings for the test_in_tmpdir.
>>>>>> I can give a coverage debug sys output from the other, original,
>>>>>> machine tomorrow, if still needed?
>>>>>>
>>>>>> Also, the code is at https://gist.github.com/bilderbuchi/6412754 (you
>>>>>> can download a tar.gz of it), if you want to try and reproduce the
>>>>>> problem.
>>>>>>
>>>>>> thanks a lot,
>>>>>> Christoph
>>>>> Christoph, instead of spending time reproducing your problem, I tried
>>>>> adding a feature to coverage.py to help you diagnose it. Attached is a
>>>>> tarball of the coverage.py kit.  It adds a flag:
>>>>>
>>>>>       coverage run --debug=notrace  myprog.py etc...
>>>>>
>>>>> This will make coverage.py write to stderr for every file it decides
>>>>> not
>>>>> to trace.  Each file is listed with the reason it isn't being traced.
>>>>> Can you try your scenario with it, and see if it gives you useful
>>>>> information?
>>>>>
>>>>> --Ned.
>>>>>
>>>> Ned,
>>>> thanks for the patched version! Unfortunately, I can't seem to
>>>> trigger that option from a .coveragerc containing
>>>> [run]
>>>> debug = notrace
>>>> when doing $py.test --cov-config=.coveragerc --cov=my_script.py
>>>> tests/test_in_tmpdir.py
>>>> (it works in coverage itself cause I see the output when using the
>>>> `coverage` command. Doesn't show anything useful though due to not
>>>> being modded for subprocess tracking yet)
>>>> I have been using pytest-cov so far, which has subprocess covering
>>>> out-of-the-box. I am a bit hesitant fiddling with my (working) python
>>>> environment.
>>>> Also, pytest-cov is proven to work(!) with the test_in_scriptdir
>>>> test, where I get coverage, so a general malfunction there can be
>>>> excluded.
>>>>
>>>> Reproducing my problem would be easy, though:
>>>> * download the files, put the appropriate ones into a tests/
>>>> directory as noted in the comments within (github gists sadly don't
>>>> offer subdirectories)
>>>> * install pytest-cov if necessary
>>>> * run the commands from the log file I attached to my previous mail
>>>> * see if you get the same problem with the test_in_tmpdir.
>>>>
>>>> Anything further I can do to diagnose this?
>>>> I can later try modifying my python install according to the guide to
>>>> make vanilla coverage work, to be able to use your patched version to
>>>> reproduce.
>>>>
>>>> thanks,
>>>> Christoph
>>>>
>>>> _______________________________________________
>>>> testing-in-python mailing list
>>>> testing-in-python at lists.idyll.org
>>>> http://lists.idyll.org/listinfo/testing-in-python
>>> OK, after some further investigations on my home system, where I have
>>> enticed vanilla coverage to cover subprocesses (going the .pth route),
>>> with your patched version, it finds the script file for both tests,
>>> but doesn't cover it in case of test_tmpdir. See attached logfile
>>> console_log2.txt.
>>>
>>> best,
>>> many thanks for any help,
>>> christoph
>> Christoph, my best guess is that pytest-cov doesn't properly account for
>> the tmpdir feature of py.test.  When you provide --cov=my_script.py,
>> pytest-cov passes that to coverage.py as the "source" parameter.
>> Coverage.py will not measure code outside of "source", so when you run
>> the test in a tmpdir, the code being run has been copied outside of the
>> location coverage.py was told to focus on, and so coverage.py refuses to
>> measure it.
>>
>> I don't understand why my_script.py doesn't appear in the debug=notrace
>> output, perhaps that setting isn't being used in the subprocess?
>>
>> Can you try the tests again without the --cov switch, or is that the
>> only way to request coverage measurement?
>>
>> --Ned.
>>
> Ned,
>
> The concise summary:
> I think it's possible the core of the problem is that when my_script.py is called _from another directory_ via <path-to-script>/my_script.py, coverage for some reason (some path assumptions maybe?) does not pick up on covered lines, although it recognizes the existence of my_script.py itself correctly.
>
> The detailed version:
> There's several problems here that I see:
> #1: pytest-cov doesn't seem to see my_script when I run the tmpdir test. It's difficult to debug why this happens, because of
> #2: pytest-cov does not seem to correctly parse the debug=notrace option when given in a .coveragerc file, leading to no debug output (as seen at the end of console_log2.txt)
>
> To exclude any influence and/or bugs within pytest-cov (so we don't get distracted by them, let's forget about pytest-cov for a while), I have run coverage itself (not via `py.test --cov`), as shown in console_log2.txt.
> The first test run with the script_dir test shows that subprocess coverage works.
> The second test run with the tmpdir_test exposes
> #3: coverage picks up my_script, but it shows 0% coverage (although the test runs the script correctly)!
> So #3 is a bug in coverage I think, as there's no pytest-cov involved at all. Possibly #1 is triggered by #3 and would go away if #3 is fixed.
>
>> Coverage.py will not measure code outside of "source", so when you run
>> the test in a tmpdir, the code being run has been copied outside of the
>> location coverage.py was told to focus on, and so coverage.py refuses to
>> measure it.
> This is not correct. The only file being copied to tmpdir is arg_file.txt. The script stays in place.
> The only difference between the two tests is that once my_script.py called from within the script directory with
> ./my_script.py <path-to-tmp>/arg_file.txt (this works)
> and in the other test it's called from within the tmpdir directory with
> <path-to-script>/my_script.py arg_file.txt
> and it's the latter test that doesn't show coverage on my_script.py.
>
>> I don't understand why my_script.py doesn't appear in the debug=notrace
>> output, perhaps that setting isn't being used in the subprocess?
> It doesn't appear in the pure coverage calls because it has been picked up, just shows 0 coverage (as evidenced by "my_script 3 3 0%" in L283 of console_log2.txt)
> It doesn't appear in the py.test --cov calls because here the debug option does not seem to be picked up, as there's no debug output at all (see L291 ff.)
>
>> Can you try the tests again without the --cov switch, or is that the
>> only way to request coverage measurement?
> Without --cov, py.test only runs the test, no coverage reporting is done. But as I said, let's focus on pure coverage (i.e. #3), and leave pytest-cov out of this for now.
>
> all the best,
> Christoph

OK, can you provide me with detailed instructions and ideally a tarball 
of files that I can use to reproduce the problem?

--Ned.