[TIP] Coverage does not cover script if py.test executes it from another directory

[Ned Batchelder]

On 9/2/13 5:26 PM, Christoph Buchner wrote:
>
> On 09/02/2013 07:13 PM, Ned Batchelder wrote:
>> On 9/2/13 11:45 AM, Christoph Buchner wrote:
>>> Hi folks,
>>>
>>> I have run into an issue using py.test, pytest-cov and coverage. The
>>> question is also on Stackoverflow in all detail
>>> (http://stackoverflow.com/q/18573542/599884), but I'll also
>>> summarize it here:
>>>
>>> I'm writing a python script which takes command line arguments, 
>>> working with
>>> some files (git repos etc).
>>> I've written succeeding tests with py.test putting this script 
>>> through its
>>> paces (verifying it works correctly with different arguments, etc),
>>> executing it with subprocess.call in the tests.
>>>
>>> Now I want to analyze code coverage with coverage.py.
>>> Coverage, when used via the pytest-cov plugin (which has 
>>> subprocess-handling
>>> already built-in/ootb), does not see/cover my script when the script is
>>> called from a temporary testing directory created with py.test's tmpdir
>>> fixture.
>>> Coverage does see my script when it's called in the directory it 
>>> resides in
>>> (and the filename argument points to a remote path).
>>>
>>> To illustrate with a minimal example,
>>>> Running in directory /home/cbuchner/pytest_experiment
>>>> Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
>>> gets correct coverage measurement, but
>>>> Running in directory /tmp/pytest-52/test_11
>>>> Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
>>> gives "Coverage.py warning: No data was collected."
>>> See http://stackoverflow.com/q/18573542/599884 for full
>>> console output and link to the used code.
>>>
>>> In both situations, my tests pass! This is on Ubuntu, with Coverage 
>>> 3.6,
>>> pytest-2.3.5, pytest-cov 1.6, all from PyPi.
>>>
>>> Questions:
>>> * How can I get coverage to recognize my script even if it's 
>>> executed in
>>> another directory?
>>> * Is this a bug in coverage, or something which is just not possible 
>>> to do?
>>> Would be surprised if the latter, after all, tmpdir is a stock 
>>> mechanism of
>>> py.test, right?
>>> * If the way I did it is an inappropriate testing implementation, 
>>> I'd be
>>> grateful for pointers how to "properly" coverage-test file-manipulating
>>> console utilities written in Python. Writing automated tests is 
>>> pretty new
>>> to me, so I'm learning much and know little (yet).
>>>
>>> thanks for any pointers on this,
>>> best,
>>> Christoph
>>>
>> It's a little hard to know why it's not working without more details, 
>> but a first thing that comes to mind is that coverage ignores code it 
>> thinks is in the stdlib.  If your script is being run from a 
>> directory under the stdlib somehow, then that could explain it.
>>
>> Can you run "coverage debug sys" in the same context as your failing 
>> scenario?  It will show details of the configuration.
>>
>> --Ned.
> Yes, that's why I gave as much detail as possible (including code) in 
> the SO answer. Didn't think of coverage debug sys, though, sorry. >.<
> I'm at another machine now, which exhibits the same problem.
> Find attached a console log (it's too long for an email I think). I 
> don't think the paths I'm working in are under the stdlib. At the end 
> of the log, I also ran the py.test with coverage with "-s", too, and 
> get two additional warnings for the test_in_tmpdir.
> I can give a coverage debug sys output from the other, original, 
> machine tomorrow, if still needed?
>
> Also, the code is at https://gist.github.com/bilderbuchi/6412754 (you 
> can download a tar.gz of it), if you want to try and reproduce the 
> problem.
>
> thanks a lot,
> Christoph

Christoph, instead of spending time reproducing your problem, I tried 
adding a feature to coverage.py to help you diagnose it. Attached is a 
tarball of the coverage.py kit.  It adds a flag:

     coverage run --debug=notrace  myprog.py etc...

This will make coverage.py write to stderr for every file it decides not 
to trace.  Each file is listed with the reason it isn't being traced.  
Can you try your scenario with it, and see if it gives you useful 
information?

--Ned.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: coverage-3.6.1a1.tar.gz
Type: application/x-gzip
Size: 239116 bytes
Desc: not available
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20130904/1a3b5979/attachment-0001.bin>