[TIP] Coverage does not cover script if py.test executes it from another directory

Ned Batchelder ned at nedbatchelder.com
Wed Sep 4 17:03:57 PDT 2013


On 9/4/13 5:45 PM, Christoph Buchner wrote:
>
> On 09/04/2013 05:03 PM, Christoph Buchner wrote:
>> Ned Batchelder schrieb am 04.09.2013 16:11:
>>
>>> On 9/2/13 5:26 PM, Christoph Buchner wrote:
>>>> On 09/02/2013 07:13 PM, Ned Batchelder wrote:
>>>>> On 9/2/13 11:45 AM, Christoph Buchner wrote:
>>>>>> Hi folks,
>>>>>>
>>>>>> I have run into an issue using py.test, pytest-cov and coverage. The
>>>>>> question is also on Stackoverflow in all detail
>>>>>> (http://stackoverflow.com/q/18573542/599884), but I'll also
>>>>>> summarize it here:
>>>>>>
>>>>>> I'm writing a python script which takes command line arguments,
>>>>>> working with
>>>>>> some files (git repos etc).
>>>>>> I've written succeeding tests with py.test putting this script
>>>>>> through its
>>>>>> paces (verifying it works correctly with different arguments, etc),
>>>>>> executing it with subprocess.call in the tests.
>>>>>>
>>>>>> Now I want to analyze code coverage with coverage.py.
>>>>>> Coverage, when used via the pytest-cov plugin (which has
>>>>>> subprocess-handling
>>>>>> already built-in/ootb), does not see/cover my script when the 
>>>>>> script is
>>>>>> called from a temporary testing directory created with py.test's 
>>>>>> tmpdir
>>>>>> fixture.
>>>>>> Coverage does see my script when it's called in the directory it
>>>>>> resides in
>>>>>> (and the filename argument points to a remote path).
>>>>>>
>>>>>> To illustrate with a minimal example,
>>>>>>> Running in directory /home/cbuchner/pytest_experiment
>>>>>>> Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
>>>>>> gets correct coverage measurement, but
>>>>>>> Running in directory /tmp/pytest-52/test_11
>>>>>>> Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
>>>>>> gives "Coverage.py warning: No data was collected."
>>>>>> See http://stackoverflow.com/q/18573542/599884 for full
>>>>>> console output and link to the used code.
>>>>>>
>>>>>> In both situations, my tests pass! This is on Ubuntu, with Coverage
>>>>>> 3.6,
>>>>>> pytest-2.3.5, pytest-cov 1.6, all from PyPi.
>>>>>>
>>>>>> Questions:
>>>>>> * How can I get coverage to recognize my script even if it's
>>>>>> executed in
>>>>>> another directory?
>>>>>> * Is this a bug in coverage, or something which is just not possible
>>>>>> to do?
>>>>>> Would be surprised if the latter, after all, tmpdir is a stock
>>>>>> mechanism of
>>>>>> py.test, right?
>>>>>> * If the way I did it is an inappropriate testing implementation,
>>>>>> I'd be
>>>>>> grateful for pointers how to "properly" coverage-test 
>>>>>> file-manipulating
>>>>>> console utilities written in Python. Writing automated tests is
>>>>>> pretty new
>>>>>> to me, so I'm learning much and know little (yet).
>>>>>>
>>>>>> thanks for any pointers on this,
>>>>>> best,
>>>>>> Christoph
>>>>>>
>>>>> It's a little hard to know why it's not working without more details,
>>>>> but a first thing that comes to mind is that coverage ignores code it
>>>>> thinks is in the stdlib.  If your script is being run from a
>>>>> directory under the stdlib somehow, then that could explain it.
>>>>>
>>>>> Can you run "coverage debug sys" in the same context as your failing
>>>>> scenario?  It will show details of the configuration.
>>>>>
>>>>> --Ned.
>>>> Yes, that's why I gave as much detail as possible (including code) in
>>>> the SO answer. Didn't think of coverage debug sys, though, sorry. >.<
>>>> I'm at another machine now, which exhibits the same problem.
>>>> Find attached a console log (it's too long for an email I think). I
>>>> don't think the paths I'm working in are under the stdlib. At the end
>>>> of the log, I also ran the py.test with coverage with "-s", too, and
>>>> get two additional warnings for the test_in_tmpdir.
>>>> I can give a coverage debug sys output from the other, original,
>>>> machine tomorrow, if still needed?
>>>>
>>>> Also, the code is at https://gist.github.com/bilderbuchi/6412754 (you
>>>> can download a tar.gz of it), if you want to try and reproduce the
>>>> problem.
>>>>
>>>> thanks a lot,
>>>> Christoph
>>> Christoph, instead of spending time reproducing your problem, I tried
>>> adding a feature to coverage.py to help you diagnose it. Attached is a
>>> tarball of the coverage.py kit.  It adds a flag:
>>>
>>>      coverage run --debug=notrace  myprog.py etc...
>>>
>>> This will make coverage.py write to stderr for every file it decides 
>>> not
>>> to trace.  Each file is listed with the reason it isn't being traced.
>>> Can you try your scenario with it, and see if it gives you useful
>>> information?
>>>
>>> --Ned.
>>>
>> Ned,
>> thanks for the patched version! Unfortunately, I can't seem to 
>> trigger that option from a .coveragerc containing
>> [run]
>> debug = notrace
>> when doing $py.test --cov-config=.coveragerc --cov=my_script.py 
>> tests/test_in_tmpdir.py
>> (it works in coverage itself cause I see the output when using the 
>> `coverage` command. Doesn't show anything useful though due to not 
>> being modded for subprocess tracking yet)
>> I have been using pytest-cov so far, which has subprocess covering 
>> out-of-the-box. I am a bit hesitant fiddling with my (working) python 
>> environment.
>> Also, pytest-cov is proven to work(!) with the test_in_scriptdir 
>> test, where I get coverage, so a general malfunction there can be 
>> excluded.
>>
>> Reproducing my problem would be easy, though:
>> * download the files, put the appropriate ones into a tests/ 
>> directory as noted in the comments within (github gists sadly don't 
>> offer subdirectories)
>> * install pytest-cov if necessary
>> * run the commands from the log file I attached to my previous mail
>> * see if you get the same problem with the test_in_tmpdir.
>>
>> Anything further I can do to diagnose this?
>> I can later try modifying my python install according to the guide to 
>> make vanilla coverage work, to be able to use your patched version to 
>> reproduce.
>>
>> thanks,
>> Christoph
>>
>> _______________________________________________
>> testing-in-python mailing list
>> testing-in-python at lists.idyll.org
>> http://lists.idyll.org/listinfo/testing-in-python
> OK, after some further investigations on my home system, where I have 
> enticed vanilla coverage to cover subprocesses (going the .pth route), 
> with your patched version, it finds the script file for both tests, 
> but doesn't cover it in case of test_tmpdir. See attached logfile 
> console_log2.txt.
>
> best,
> many thanks for any help,
> christoph

Christoph, my best guess is that pytest-cov doesn't properly account for 
the tmpdir feature of py.test.  When you provide --cov=my_script.py, 
pytest-cov passes that to coverage.py as the "source" parameter.  
Coverage.py will not measure code outside of "source", so when you run 
the test in a tmpdir, the code being run has been copied outside of the 
location coverage.py was told to focus on, and so coverage.py refuses to 
measure it.

I don't understand why my_script.py doesn't appear in the debug=notrace 
output, perhaps that setting isn't being used in the subprocess?

Can you try the tests again without the --cov switch, or is that the 
only way to request coverage measurement?

--Ned.



More information about the testing-in-python mailing list