[TIP] Coverage does not cover script if py.test executes it from another directory
Christoph Buchner
bilderbuchi at phononoia.at
Mon Sep 2 14:26:51 PDT 2013
On 09/02/2013 07:13 PM, Ned Batchelder wrote:
> On 9/2/13 11:45 AM, Christoph Buchner wrote:
>> Hi folks,
>>
>> I have run into an issue using py.test, pytest-cov and coverage. The
>> question is also on Stackoverflow in all detail
>> (http://stackoverflow.com/q/18573542/599884), but I'll also
>> summarize it here:
>>
>> I'm writing a python script which takes command line arguments,
>> working with
>> some files (git repos etc).
>> I've written succeeding tests with py.test putting this script
>> through its
>> paces (verifying it works correctly with different arguments, etc),
>> executing it with subprocess.call in the tests.
>>
>> Now I want to analyze code coverage with coverage.py.
>> Coverage, when used via the pytest-cov plugin (which has
>> subprocess-handling
>> already built-in/ootb), does not see/cover my script when the script is
>> called from a temporary testing directory created with py.test's tmpdir
>> fixture.
>> Coverage does see my script when it's called in the directory it
>> resides in
>> (and the filename argument points to a remote path).
>>
>> To illustrate with a minimal example,
>>> Running in directory /home/cbuchner/pytest_experiment
>>> Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
>> gets correct coverage measurement, but
>>> Running in directory /tmp/pytest-52/test_11
>>> Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
>> gives "Coverage.py warning: No data was collected."
>> See http://stackoverflow.com/q/18573542/599884 for full
>> console output and link to the used code.
>>
>> In both situations, my tests pass! This is on Ubuntu, with Coverage 3.6,
>> pytest-2.3.5, pytest-cov 1.6, all from PyPi.
>>
>> Questions:
>> * How can I get coverage to recognize my script even if it's executed in
>> another directory?
>> * Is this a bug in coverage, or something which is just not possible
>> to do?
>> Would be surprised if the latter, after all, tmpdir is a stock
>> mechanism of
>> py.test, right?
>> * If the way I did it is an inappropriate testing implementation, I'd be
>> grateful for pointers how to "properly" coverage-test file-manipulating
>> console utilities written in Python. Writing automated tests is
>> pretty new
>> to me, so I'm learning much and know little (yet).
>>
>> thanks for any pointers on this,
>> best,
>> Christoph
>>
> It's a little hard to know why it's not working without more details,
> but a first thing that comes to mind is that coverage ignores code it
> thinks is in the stdlib. If your script is being run from a directory
> under the stdlib somehow, then that could explain it.
>
> Can you run "coverage debug sys" in the same context as your failing
> scenario? It will show details of the configuration.
>
> --Ned.
Yes, that's why I gave as much detail as possible (including code) in
the SO answer. Didn't think of coverage debug sys, though, sorry. >.<
I'm at another machine now, which exhibits the same problem.
Find attached a console log (it's too long for an email I think). I
don't think the paths I'm working in are under the stdlib. At the end of
the log, I also ran the py.test with coverage with "-s", too, and get
two additional warnings for the test_in_tmpdir.
I can give a coverage debug sys output from the other, original, machine
tomorrow, if still needed?
Also, the code is at https://gist.github.com/bilderbuchi/6412754 (you
can download a tar.gz of it), if you want to try and reproduce the problem.
thanks a lot,
Christoph
-------------- next part --------------
bilderbuchi at renderbiest:/media/windata/Visuals/Coding/pytest_experiment$ py.test -s
======================================================= test session starts ========================================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 2 items
tests/test_in_scriptdir.py
set_up: In directory /tmp/pytest-0/test_10
Running in directory /media/windata/Visuals/Coding/pytest_experiment
Command: ./my_script.py /tmp/pytest-0/test_10/arg_file.txt
--Contents of arg_file.txt--
.
tests/test_in_tmpdir.py
set_up: In directory /tmp/pytest-0/test_11
Running in directory /tmp/pytest-0/test_11
Command: /media/windata/Visuals/Coding/pytest_experiment/my_script.py arg_file.txt
--Contents of arg_file.txt--
.
===================================================== 2 passed in 0.12 seconds =====================================================
bilderbuchi at renderbiest:/media/windata/Visuals/Coding/pytest_experiment$ py.test --cov=my_script.py tests/test_in_scriptdir.py
======================================================= test session starts ========================================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items
tests/test_in_scriptdir.py .
----------------------------------------- coverage: platform linux2, python 2.7.4-final-0 ------------------------------------------
Name Stmts Miss Cover
-------------------------------
my_script 3 0 100%
===================================================== 1 passed in 0.10 seconds =====================================================
bilderbuchi at renderbiest:/media/windata/Visuals/Coding/pytest_experiment$ py.test --cov=my_script.py tests/test_in_tmpdir.py
======================================================= test session starts ========================================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items
tests/test_in_tmpdir.py .Coverage.py warning: No data was collected.
----------------------------------------- coverage: platform linux2, python 2.7.4-final-0 ------------------------------------------
Name Stmts Miss Cover
---------------------------
===================================================== 1 passed in 0.09 seconds =====================================================
bilderbuchi at renderbiest:/media/windata/Visuals/Coding/pytest_experiment$ coverage debug sys
-- sys ----------------------------------------
version: 3.6
coverage: /home/bilderbuchi/.local/lib/python2.7/site-packages/coverage/__init__.pyc
cover_dir: /home/bilderbuchi/.local/lib/python2.7/site-packages/coverage
pylib_dirs: /usr/lib/python2.7
tracer: CTracer
config_files: .coveragerc
configs_read: -none-
data_path: /media/windata/Visuals/Coding/pytest_experiment/.coverage
python: 2.7.4 (default, Apr 19 2013, 18:28:01) [GCC 4.7.3]
platform: Linux-3.8.0-29-generic-x86_64-with-Ubuntu-13.04-raring
implementation: CPython
executable: /usr/bin/python
cwd: /media/windata/Visuals/Coding/pytest_experiment
path: /home/bilderbuchi/.local/bin
/usr/local/lib/python2.7/dist-packages/times-0.4-py2.7.egg
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/home/bilderbuchi/.local/lib/python2.7/site-packages
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages/PILcompat
/usr/lib/python2.7/dist-packages/gst-0.10
/usr/lib/python2.7/dist-packages/gtk-2.0
/usr/lib/pymodules/python2.7
/usr/lib/python2.7/dist-packages/ubuntu-sso-client
/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode
environment: -none-
bilderbuchi at renderbiest:/media/windata/Visuals/Coding/pytest_experiment$ cd /tmp/pytest-0/test_10/
bilderbuchi at renderbiest:/tmp/pytest-0/test_10$ coverage debug sys
-- sys ----------------------------------------
version: 3.6
coverage: /home/bilderbuchi/.local/lib/python2.7/site-packages/coverage/__init__.pyc
cover_dir: /home/bilderbuchi/.local/lib/python2.7/site-packages/coverage
pylib_dirs: /usr/lib/python2.7
tracer: CTracer
config_files: .coveragerc
configs_read: -none-
data_path: /tmp/pytest-0/test_10/.coverage
python: 2.7.4 (default, Apr 19 2013, 18:28:01) [GCC 4.7.3]
platform: Linux-3.8.0-29-generic-x86_64-with-Ubuntu-13.04-raring
implementation: CPython
executable: /usr/bin/python
cwd: /tmp/pytest-0/test_10
path: /home/bilderbuchi/.local/bin
/usr/local/lib/python2.7/dist-packages/times-0.4-py2.7.egg
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/home/bilderbuchi/.local/lib/python2.7/site-packages
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages/PILcompat
/usr/lib/python2.7/dist-packages/gst-0.10
/usr/lib/python2.7/dist-packages/gtk-2.0
/usr/lib/pymodules/python2.7
/usr/lib/python2.7/dist-packages/ubuntu-sso-client
/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode
environment: -none-
bilderbuchi at renderbiest:/tmp/pytest-0/test_10$ cd /media/windata/Visuals/Coding/pytest_experiment
bilderbuchi at renderbiest:/media/windata/Visuals/Coding/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_scriptdir.py
============================================================= test session starts ==============================================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items
tests/test_in_scriptdir.py
set_up: In directory /tmp/pytest-3/test_10
Running in directory /media/windata/Visuals/Coding/pytest_experiment
Command: ./my_script.py /tmp/pytest-3/test_10/arg_file.txt
--Contents of arg_file.txt--
.
----------------------------------------------- coverage: platform linux2, python 2.7.4-final-0 ------------------------------------------------
Name Stmts Miss Cover
-------------------------------
my_script 3 0 100%
=========================================================== 1 passed in 0.12 seconds ===========================================================
bilderbuchi at renderbiest:/media/windata/Visuals/Coding/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_tmpdir.py
============================================================= test session starts ==============================================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items
tests/test_in_tmpdir.py
set_up: In directory /tmp/pytest-4/test_10
Running in directory /tmp/pytest-4/test_10
Command: /media/windata/Visuals/Coding/pytest_experiment/my_script.py arg_file.txt
--Contents of arg_file.txt--
Coverage.py warning: Module my_script.py was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: No data was collected.
----------------------------------------------- coverage: platform linux2, python 2.7.4-final-0 ------------------------------------------------
Name Stmts Miss Cover
---------------------------
=========================================================== 1 passed in 0.13 seconds ===========================================================
bilderbuchi at renderbiest:/media/windata/Visuals/Coding/pytest_experiment$
More information about the testing-in-python
mailing list