[TIP] coverage.py against a server
Ned Batchelder
ned at nedbatchelder.com
Fri Sep 27 19:25:18 PDT 2013
On 9/27/13 8:01 AM, Christoph Buchner wrote:
>
> Ned Batchelder schrieb am 27.09.2013 13:27:
>
>> On 9/25/13 5:27 AM, Christoph Buchner wrote:
>>> Ned Batchelder schrieb am 20.09.2013 18:57:
>>>
>>>> On 9/19/13 3:58 AM, Christoph Buchner wrote:
>>>>> re this:
>>>>>> Right, --source trumps --include. You've already told coverage that
>>>>>> the
>>>>>> only interesting files are in ../src, so there's nothing else to
>>>>>> include. I should make a warning for inconsistent options like this
>>>>>> (ticket:
>>>>>> https://bitbucket.org/ned/coveragepy/issue/265/when-using-source-include-is-silently)
>>>>> are you sure this is 100% correct? I think there' more nuances here. I
>>>>> have observed during working with my remote-executing script (mail
>>>>> convo
>>>>> up until a couple days ago), that there's a difference in behaviour
>>>>> between only source, and both source and include being specified.
>>>> Christoph, reading the coverage.py code, it seems clear to me that when
>>>> --source is specified, --include is ignored. I could be mis-reading the
>>>> code, though. Can you re-run your tests, and show the results? The
>>>> results you describe for the source-but-no-include case seems wrong: it
>>>> isn't even limiting coverage to the --source directory.
>>>>
>>>> --Ned.
>>> Ned, sorry for the delay. Using the repo commit
>>> https://github.com/bilderbuchi/ofStateManager/tree/5a61f2159fe6d4da6b3a60521e9e62c82b634cae
>>> I get the following log output demonstrating this behaviour:
>>> https://gist.github.com/bilderbuchi/6697184
>> Christoph, I tried the repo, every test failed with a Permission Denied
>> error, perhaps because I don't have Open Framework installed? In any
>> case, I created a test file in your tests directory, and saw the
>> behavior you are describing.
> Ned,
> Permission denied was because ofStateManager was not marked executable. Tahnks for catching this! This slipped through the cracks because I normally developed on an NTFs partition on a linux/windows dualboot machine, where everything is mounted executable. I've corrected this, should run now: https://github.com/bilderbuchi/ofStateManager/tree/96669dfe96ba4bcd866a1ec9eaf6204e2c84051a
>
> OpenFrameworks does not need to be installed, all the testing repos are inside the tar.gz file in /tests, and extracted for testing.
>
>> The full story is that when running code, --source overrides --include,
>> and if both are specified, --include is silently ignored. When
>> reporting, --source is never used at all, and --include takes effect.
>> Yes, this is confusing, and something should be changed.
>>
> I see, thanks for clearing this up.
>
>> I now have two tickets:
>> https://bitbucket.org/ned/coveragepy/issue/266/report-command-doesnt-accept-source-option
>> https://bitbucket.org/ned/coveragepy/issue/265/when-using-source-include-is-silently
>>
>> Advice gratefully accepted.
>>
>> BTW: You are doing a lot of work in run_coverage.py that would be much
>> easier with a shell script.
> Yeah I didn't want to use a shell script for this because I'm much more familiar with python than with bash.
>
>> You are configuring a coverage object, but
>> then spawning a subprocess to run coverage. So for example, your
>> --source specification is never used at all, which is why you get
>> reporting on all sorts of code outside your project in the second case.
> Ah interesting. I did it that way because I didn't know better, and I thought that, when I'm already writing a python script, I might as well use the API. :-P
> Am I just missing a cov.start(), cov.stop() around the subprocess call? Can you tell me how to "properly" to the same thing as the subprocess call using the coverage API?
The API calls you are making are configuring coverage measurement in
your main process. Then you use a subprocess to run your tests. The two
are independent. No amount of API calling in the main process will
change how coverage works in the subprocess. If you want to provide a
--source argument to your subprocess, then either add it to the
sub-process's command line, or put it in the .coveragerc.
--Ned.
>
>> Also, you specify explicit values for things like the data file and the
>> rc file, but those values are exactly the default values. None of this
>> affects the source/include issue at hand, but you could simplify your
>> testing harness.
> This is because of the problem with calling the script from a remote test directory, as we discussed earlier. That's why I need absolute paths in the environment variables, otherwise the coverage doesn't get picked up correctly. That is why I specify the paths by hand.
>
> thanks for everything,
> christoph
>
>> --Ned.
>>> best,
>>> thanks,
>>> christoph
>>>
>>>>> Specifically, using this script to run coverage:
>>>>>
>>>>> #!/usr/bin/env python
>>>>> import os
>>>>> import sys
>>>>> import inspect
>>>>> import coverage
>>>>> import subprocess
>>>>>
>>>>> arguments = ''
>>>>> if len(sys.argv) > 1:
>>>>> arguments = ' '.join(sys.argv[1:])
>>>>>
>>>>> testdir =
>>>>> os.path.abspath(os.path.dirname(inspect.getfile(inspect.currentframe())))
>>>>> os.environ['COVERAGE_PROCESS_START'] = os.path.join(testdir,
>>>>> '.coveragerc')
>>>>> os.environ['COVERAGE_FILE'] = os.path.join(testdir, '.coverage')
>>>>> cov = coverage.coverage(source=os.path.join(testdir, '..'),
>>>>> include=os.path.join(testdir, '..', 'ofStateManager.py'))
>>>>>
>>>>> cov.erase()
>>>>> subprocess.call('coverage run -m py.test ' + arguments,
>>>>> shell=True,
>>>>> cwd=testdir)
>>>>> cov.combine()
>>>>> cov.html_report(directory=os.path.join(testdir, 'htmlcov'))
>>>>> cov.report(show_missing=False)
>>>>>
>>>>> If I run this as-is, the report contains only ofStateManager.py, as
>>>>> desired.
>>>>> If I strip out the include=... part in coverage.coverage(...), I get a
>>>>> coverage report on ofStateManager.py, the tests I run, and a load of
>>>>> things in ~/.local/lib/python2.7/site-packages/ and
>>>>> /usr/share/pyshared/,
>>>>> which I don't want. This was the reason to add that include= option in
>>>>> the first place.
>>>>> (fwiw, the whole thing lives at
>>>>> https://github.com/bilderbuchi/ofStateManager/tree/tests currently)
>>>>>
>>>>> best,
>>>>> christoph
>>>>>
>>>>>
>>>>> Ned Batchelder schrieb am 19.09.2013 01:43:
>>>>>
>>>>>> On 9/18/13 9:10 AM, Lucian Ciufudean wrote:
>>>>>>> Sorry for all these iterations, here is a more consistent (but long)
>>>>>>> email.
>>>>>>>
>>>>>> Iterations are fine as long as they bring more detail! Thanks for
>>>>>> putting in all the work.
>>>>>>
>>>>>>> After not being able to run 'coverage run pyc_file.pyc', I created a
>>>>>>> dummy driver main.py file that uses the compiled modules.
>>>>>>>
>>>>>>> root at spahire pyc]# ls -la
>>>>>>> total 20
>>>>>>> drwxr-xr-x. 2 root root 4096 Sep 18 13:20 .
>>>>>>> drwxr-xr-x. 4 root root 4096 Sep 17 15:58 ..
>>>>>>> -rw-r--r--. 1 root root 28 Sep 18 13:01 main.py
>>>>>>> -rw-------. 1 root root 187 Sep 17 15:59 test_coverage_callee.pyc
>>>>>>> -rw-------. 1 root root 212 Sep 17 15:59 test_coverage_caller.pyc
>>>>>>> [root at spahire pyc]# cat main.py
>>>>>>> import test_coverage_caller
>>>>>>>
>>>>>>> The source files for the 2 pyc files are here:
>>>>>>>
>>>>>>> [root at spahire pyc]# ls ../src
>>>>>>> test_coverage_callee.py test_coverage_caller.py
>>>>>>>
>>>>>>> Further, I run coverage, I get a warning but a data file .coverage is
>>>>>>> created:
>>>>>>>
>>>>>>> [root at spahire pyc]# coverage run --source=../src main.py
>>>>>>> 1
>>>>>>> 2
>>>>>>> 2.1
>>>>>>> 2.2
>>>>>>> 2.3
>>>>>>> Coverage.py warning: No data was collected.
>>>>>>> [root at spahire pyc]# ls -l .coverage
>>>>>>> -rw-r--r--. 1 root root 180 Sep 18 13:23 .coverage
>>>>>>>
>>>>>> The --source option tells coverage that the only files of interest are
>>>>>> the ones in ../src. You never execute any files in ../src, so
>>>>>> coverage
>>>>>> hasn't collected any data.
>>>>>>
>>>>>>> --include does not make a difference:
>>>>>>>
>>>>>>> [root at spahire pyc]# coverage run --source=../src --include='*'
>>>>>>> main.py
>>>>>>> 1
>>>>>>> 2
>>>>>>> 2.1
>>>>>>> 2.2
>>>>>>> 2.3
>>>>>>> Coverage.py warning: No data was collected.
>>>>>>> [root at spahire pyc]# ls -l .coverage
>>>>>>> -rw-r--r--. 1 root root 180 Sep 18 13:38 .coverage
>>>>>>>
>>>>>> Right, --source trumps --include. You've already told coverage that
>>>>>> the
>>>>>> only interesting files are in ../src, so there's nothing else to
>>>>>> include. I should make a warning for inconsistent options like this
>>>>>> (ticket:
>>>>>> https://bitbucket.org/ned/coveragepy/issue/265/when-using-source-include-is-silently)
>>>>>>
>>>>>>> As expected, report shows nothing is covered:
>>>>>>>
>>>>>>> [root at spahire pyc]# coverage report --include='*test*'
>>>>>>> Name Stmts Miss Cover
>>>>>>> ----------------------------------------------------------------------------
>>>>>>> /root/lucian/coverage/module1/src/test_coverage_callee 3 3
>>>>>>> 0%
>>>>>>> /root/lucian/coverage/module1/src/test_coverage_caller 3 3
>>>>>>> 0%
>>>>>>> ----------------------------------------------------------------------------
>>>>>>> TOTAL 6 6 0%
>>>>>>>
>>>>>>> report does not accept --source, might this be the root to all evil?
>>>>>>>
>>>>>> Hmm, that seems like an oversight: I should make that possible
>>>>>> (ticket:
>>>>>> https://bitbucket.org/ned/coveragepy/issue/266/report-command-doesnt-accept-source-option)
>>>>>>> [root at spahire pyc]# coverage report --include='*test*'
>>>>>>> --source=../src
>>>>>>> no such option: --source
>>>>>>> Use 'coverage help' for help.
>>>>>>>
>>>>>>> No warning when using run without options, but the report is again
>>>>>>> wrong:
>>>>>>>
>>>>>>> [root at spahire pyc]# coverage run main.py
>>>>>>> 1
>>>>>>> 2
>>>>>>> 2.1
>>>>>>> 2.2
>>>>>>> 2.3
>>>>>>> [root at spahire pyc]# coverage report
>>>>>>> Name Stmts Miss Cover
>>>>>>> ------------------------------------------
>>>>>>> main 1 0 100%
>>>>>>> test_coverage_callee NoSource: No source for code:
>>>>>>> '/root/lucian/coverage/module1/pyc/test_coverage_callee.py'
>>>>>>> test_coverage_caller NoSource: No source for code:
>>>>>>> '/root/lucian/coverage/module1/pyc/test_coverage_caller.py'
>>>>>>>
>>>>>> The problem is that you haven't told coverage how to find the source
>>>>>> files that correspond to your .pyc files. The --source option doesn't
>>>>>> do that.
>>>>>>
>>>>>>> Same thing with a configuration file:
>>>>>>> [root at spahire pyc]# cat .coveragerc
>>>>>>> [run]
>>>>>>> source=../src/
>>>>>>> include=*
>>>>>>>
>>>>>> Right, same options, specified in a different way.
>>>>>>> So now I turn to the api + the same .coveragerc, the same thing:
>>>>>>>
>>>>>>> [root at spahire pyc]# cat api.py
>>>>>>> import coverage
>>>>>>> cov = coverage.coverage()
>>>>>>> cov.start()
>>>>>>> import test_coverage_caller
>>>>>>> cov.stop()
>>>>>>> cov.save()
>>>>>>>
>>>>>>> [root at spahire pyc]# python api.py
>>>>>>> 1
>>>>>>> 2
>>>>>>> 2.1
>>>>>>> 2.2
>>>>>>> 2.3
>>>>>>> Coverage.py warning: No data was collected.
>>>>>>>
>>>>>> Another run with the same (non-)options.
>>>>>>
>>>>>>> As for your suggestion with [paths], the docs suggest to me that this
>>>>>>> is for combining data.
>>>>>> Yes, it is used when combining data, I should have fleshed out my idea
>>>>>> more fully. Try creating a .coveragerc file like this:
>>>>>>
>>>>>> [run]
>>>>>> parallel = true
>>>>>>
>>>>>> [paths]
>>>>>> mysources =
>>>>>> ../src
>>>>>> .
>>>>>>
>>>>>> Then use "coverage run main.py", then "coverage combine", then
>>>>>> "coverage
>>>>>> report". If that works, we can talk about how to make it a bit
>>>>>> easier.
>>>>>>
>>>>>> Hope that helps,
>>>>>>
>>>>>> --Ned.
>>>>>>
>>>>>>> Lucian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 18, 2013 at 1:20 PM, Ned Batchelder
>>>>>>> <ned at nedbatchelder.com
>>>>>>> <mailto:ned at nedbatchelder.com>> wrote:
>>>>>>>
>>>>>>> On 9/18/13 3:42 AM, Lucian Ciufudean wrote:
>>>>>>>> Here is how I ran coverage:
>>>>>>>>
>>>>>>>> coverage run --source=../src main.py
>>>>>>>>
>>>>>>>> main.py imports a pyc file that resides in the same folder.
>>>>>>>> The
>>>>>>>> source of this pyc file is available in ../src.
>>>>>>>> (I created main.py just to go around the bug that you
>>>>>>>> submitted
>>>>>>>> above)
>>>>>>>>
>>>>>>> You don't mention the exact errors you saw (details matter),
>>>>>>> but
>>>>>>> in your previous message you mentioned the problem being while
>>>>>>> reporting. You'll also have to specify the source directory
>>>>>>> during the "coverage html" command (or coverage.html() call).
>>>>>>> A
>>>>>>> good way to do this is with a .coveragerc file.
>>>>>>>
>>>>>>> --Ned.
>>>>>>>
>>>>>>>
>>>>>>>> Lucian
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 18, 2013 at 5:05 AM, Ned Batchelder
>>>>>>>> <ned at nedbatchelder.com <mailto:ned at nedbatchelder.com>> wrote:
>>>>>>>>
>>>>>>>> On 9/17/13 7:42 AM, Lucian Ciufudean wrote:
>>>>>>>>> Hi guys,
>>>>>>>>>
>>>>>>>>> I embarked on the road of obtaining a coverage report for
>>>>>>>>> functional tests against a server process. The server is
>>>>>>>>> deployed as .pyc files, and the source files can be made
>>>>>>>>> available in a separate subversion working folder. I
>>>>>>>>> wouldn't want to edit any existing .py files.
>>>>>>>>>
>>>>>>>>> Can coverage work with .pyc files - I am getting errors
>>>>>>>>> when
>>>>>>>>> using coverage run main.pyc from the command line, so
>>>>>>>>> maybe
>>>>>>>>> with the API?
>>>>>>>> Hmm, you're right: "coverage run foo.pyc" does not work.
>>>>>>>> I've
>>>>>>>> created a ticket for this:
>>>>>>>> https://bitbucket.org/ned/coveragepy/issue/264/coverage-wont-run-pyc-files
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> I tired this also, my custom code is run but it can not
>>>>>>>>> find
>>>>>>>>> the source files at the time of coverage.html_report()
>>>>>>>>> although I passed the directory of source files to
>>>>>>>>> coverage.coverage.
>>>>>>>> You'll have to show details of how you tried to run
>>>>>>>> coverage. If the .py files are in the places reported by
>>>>>>>> your program, then it should work. You can also use the
>>>>>>>> [paths] section of a .coveragerc to instruct coverage
>>>>>>>> where
>>>>>>>> the files are.
>>>>>>>>
>>>>>>>> --Ned.
>>>>>>>>
>>>>>>>>> Thanks a lot,
>>>>>>>>> Lucian
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> testing-in-python mailing list
>>>>>>>>> testing-in-python at lists.idyll.org
>>>>>>>>> <mailto:testing-in-python at lists.idyll.org>
>>>>>>>>> http://lists.idyll.org/listinfo/testing-in-python
>>>>>> _______________________________________________
>>>>>> testing-in-python mailing list
>>>>>> testing-in-python at lists.idyll.org
>>>>>> http://lists.idyll.org/listinfo/testing-in-python
>>>>>>
More information about the testing-in-python
mailing list