[TIP] Coverage does not cover script if py.test executes it from another directory
bilderbuchi at phononoia.at
Wed Sep 4 08:03:14 PDT 2013
Ned Batchelder schrieb am 04.09.2013 16:11:
> On 9/2/13 5:26 PM, Christoph Buchner wrote:
>> On 09/02/2013 07:13 PM, Ned Batchelder wrote:
>>> On 9/2/13 11:45 AM, Christoph Buchner wrote:
>>>> Hi folks,
>>>> I have run into an issue using py.test, pytest-cov and coverage. The
>>>> question is also on Stackoverflow in all detail
>>>> (http://stackoverflow.com/q/18573542/599884), but I'll also
>>>> summarize it here:
>>>> I'm writing a python script which takes command line arguments,
>>>> working with
>>>> some files (git repos etc).
>>>> I've written succeeding tests with py.test putting this script
>>>> through its
>>>> paces (verifying it works correctly with different arguments, etc),
>>>> executing it with subprocess.call in the tests.
>>>> Now I want to analyze code coverage with coverage.py.
>>>> Coverage, when used via the pytest-cov plugin (which has
>>>> already built-in/ootb), does not see/cover my script when the script is
>>>> called from a temporary testing directory created with py.test's tmpdir
>>>> Coverage does see my script when it's called in the directory it
>>>> resides in
>>>> (and the filename argument points to a remote path).
>>>> To illustrate with a minimal example,
>>>>> Running in directory /home/cbuchner/pytest_experiment
>>>>> Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
>>>> gets correct coverage measurement, but
>>>>> Running in directory /tmp/pytest-52/test_11
>>>>> Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
>>>> gives "Coverage.py warning: No data was collected."
>>>> See http://stackoverflow.com/q/18573542/599884 for full
>>>> console output and link to the used code.
>>>> In both situations, my tests pass! This is on Ubuntu, with Coverage
>>>> pytest-2.3.5, pytest-cov 1.6, all from PyPi.
>>>> * How can I get coverage to recognize my script even if it's
>>>> executed in
>>>> another directory?
>>>> * Is this a bug in coverage, or something which is just not possible
>>>> to do?
>>>> Would be surprised if the latter, after all, tmpdir is a stock
>>>> mechanism of
>>>> py.test, right?
>>>> * If the way I did it is an inappropriate testing implementation,
>>>> I'd be
>>>> grateful for pointers how to "properly" coverage-test file-manipulating
>>>> console utilities written in Python. Writing automated tests is
>>>> pretty new
>>>> to me, so I'm learning much and know little (yet).
>>>> thanks for any pointers on this,
>>> It's a little hard to know why it's not working without more details,
>>> but a first thing that comes to mind is that coverage ignores code it
>>> thinks is in the stdlib. If your script is being run from a
>>> directory under the stdlib somehow, then that could explain it.
>>> Can you run "coverage debug sys" in the same context as your failing
>>> scenario? It will show details of the configuration.
>> Yes, that's why I gave as much detail as possible (including code) in
>> the SO answer. Didn't think of coverage debug sys, though, sorry. >.<
>> I'm at another machine now, which exhibits the same problem.
>> Find attached a console log (it's too long for an email I think). I
>> don't think the paths I'm working in are under the stdlib. At the end
>> of the log, I also ran the py.test with coverage with "-s", too, and
>> get two additional warnings for the test_in_tmpdir.
>> I can give a coverage debug sys output from the other, original,
>> machine tomorrow, if still needed?
>> Also, the code is at https://gist.github.com/bilderbuchi/6412754 (you
>> can download a tar.gz of it), if you want to try and reproduce the
>> thanks a lot,
> Christoph, instead of spending time reproducing your problem, I tried
> adding a feature to coverage.py to help you diagnose it. Attached is a
> tarball of the coverage.py kit. It adds a flag:
> coverage run --debug=notrace myprog.py etc...
> This will make coverage.py write to stderr for every file it decides not
> to trace. Each file is listed with the reason it isn't being traced.
> Can you try your scenario with it, and see if it gives you useful
thanks for the patched version! Unfortunately, I can't seem to trigger that option from a .coveragerc containing
debug = notrace
when doing $py.test --cov-config=.coveragerc --cov=my_script.py tests/test_in_tmpdir.py
(it works in coverage itself cause I see the output when using the `coverage` command. Doesn't show anything useful though due to not being modded for subprocess tracking yet)
I have been using pytest-cov so far, which has subprocess covering out-of-the-box. I am a bit hesitant fiddling with my (working) python environment.
Also, pytest-cov is proven to work(!) with the test_in_scriptdir test, where I get coverage, so a general malfunction there can be excluded.
Reproducing my problem would be easy, though:
* download the files, put the appropriate ones into a tests/ directory as noted in the comments within (github gists sadly don't offer subdirectories)
* install pytest-cov if necessary
* run the commands from the log file I attached to my previous mail
* see if you get the same problem with the test_in_tmpdir.
Anything further I can do to diagnose this?
I can later try modifying my python install according to the guide to make vanilla coverage work, to be able to use your patched version to reproduce.
More information about the testing-in-python