[TIP] coverage.py against a server
Christoph Buchner
bilderbuchi at phononoia.at
Fri Sep 27 05:01:01 PDT 2013
Ned Batchelder schrieb am 27.09.2013 13:27:
> On 9/25/13 5:27 AM, Christoph Buchner wrote:
>>
>> Ned Batchelder schrieb am 20.09.2013 18:57:
>>
>>> On 9/19/13 3:58 AM, Christoph Buchner wrote:
>>>> re this:
>>>>> Right, --source trumps --include. You've already told coverage that
>>>>> the
>>>>> only interesting files are in ../src, so there's nothing else to
>>>>> include. I should make a warning for inconsistent options like this
>>>>> (ticket:
>>>>> https://bitbucket.org/ned/coveragepy/issue/265/when-using-source-include-is-silently)
>>>> are you sure this is 100% correct? I think there' more nuances here. I
>>>> have observed during working with my remote-executing script (mail
>>>> convo
>>>> up until a couple days ago), that there's a difference in behaviour
>>>> between only source, and both source and include being specified.
>>> Christoph, reading the coverage.py code, it seems clear to me that when
>>> --source is specified, --include is ignored. I could be mis-reading the
>>> code, though. Can you re-run your tests, and show the results? The
>>> results you describe for the source-but-no-include case seems wrong: it
>>> isn't even limiting coverage to the --source directory.
>>>
>>> --Ned.
>> Ned, sorry for the delay. Using the repo commit
>> https://github.com/bilderbuchi/ofStateManager/tree/5a61f2159fe6d4da6b3a60521e9e62c82b634cae
>> I get the following log output demonstrating this behaviour:
>> https://gist.github.com/bilderbuchi/6697184
>
> Christoph, I tried the repo, every test failed with a Permission Denied
> error, perhaps because I don't have Open Framework installed? In any
> case, I created a test file in your tests directory, and saw the
> behavior you are describing.
Ned,
Permission denied was because ofStateManager was not marked executable. Tahnks for catching this! This slipped through the cracks because I normally developed on an NTFs partition on a linux/windows dualboot machine, where everything is mounted executable. I've corrected this, should run now: https://github.com/bilderbuchi/ofStateManager/tree/96669dfe96ba4bcd866a1ec9eaf6204e2c84051a
OpenFrameworks does not need to be installed, all the testing repos are inside the tar.gz file in /tests, and extracted for testing.
>
> The full story is that when running code, --source overrides --include,
> and if both are specified, --include is silently ignored. When
> reporting, --source is never used at all, and --include takes effect.
> Yes, this is confusing, and something should be changed.
>
I see, thanks for clearing this up.
> I now have two tickets:
> https://bitbucket.org/ned/coveragepy/issue/266/report-command-doesnt-accept-source-option
> https://bitbucket.org/ned/coveragepy/issue/265/when-using-source-include-is-silently
>
> Advice gratefully accepted.
>
> BTW: You are doing a lot of work in run_coverage.py that would be much
> easier with a shell script.
Yeah I didn't want to use a shell script for this because I'm much more familiar with python than with bash.
> You are configuring a coverage object, but
> then spawning a subprocess to run coverage. So for example, your
> --source specification is never used at all, which is why you get
> reporting on all sorts of code outside your project in the second case.
Ah interesting. I did it that way because I didn't know better, and I thought that, when I'm already writing a python script, I might as well use the API. :-P
Am I just missing a cov.start(), cov.stop() around the subprocess call? Can you tell me how to "properly" to the same thing as the subprocess call using the coverage API?
> Also, you specify explicit values for things like the data file and the
> rc file, but those values are exactly the default values. None of this
> affects the source/include issue at hand, but you could simplify your
> testing harness.
This is because of the problem with calling the script from a remote test directory, as we discussed earlier. That's why I need absolute paths in the environment variables, otherwise the coverage doesn't get picked up correctly. That is why I specify the paths by hand.
thanks for everything,
christoph
>
> --Ned.
>> best,
>> thanks,
>> christoph
>>
>>>> Specifically, using this script to run coverage:
>>>>
>>>> #!/usr/bin/env python
>>>> import os
>>>> import sys
>>>> import inspect
>>>> import coverage
>>>> import subprocess
>>>>
>>>> arguments = ''
>>>> if len(sys.argv) > 1:
>>>> arguments = ' '.join(sys.argv[1:])
>>>>
>>>> testdir =
>>>> os.path.abspath(os.path.dirname(inspect.getfile(inspect.currentframe())))
>>>> os.environ['COVERAGE_PROCESS_START'] = os.path.join(testdir,
>>>> '.coveragerc')
>>>> os.environ['COVERAGE_FILE'] = os.path.join(testdir, '.coverage')
>>>> cov = coverage.coverage(source=os.path.join(testdir, '..'),
>>>> include=os.path.join(testdir, '..', 'ofStateManager.py'))
>>>>
>>>> cov.erase()
>>>> subprocess.call('coverage run -m py.test ' + arguments,
>>>> shell=True,
>>>> cwd=testdir)
>>>> cov.combine()
>>>> cov.html_report(directory=os.path.join(testdir, 'htmlcov'))
>>>> cov.report(show_missing=False)
>>>>
>>>> If I run this as-is, the report contains only ofStateManager.py, as
>>>> desired.
>>>> If I strip out the include=... part in coverage.coverage(...), I get a
>>>> coverage report on ofStateManager.py, the tests I run, and a load of
>>>> things in ~/.local/lib/python2.7/site-packages/ and
>>>> /usr/share/pyshared/,
>>>> which I don't want. This was the reason to add that include= option in
>>>> the first place.
>>>> (fwiw, the whole thing lives at
>>>> https://github.com/bilderbuchi/ofStateManager/tree/tests currently)
>>>>
>>>> best,
>>>> christoph
>>>>
>>>>
>>>> Ned Batchelder schrieb am 19.09.2013 01:43:
>>>>
>>>>> On 9/18/13 9:10 AM, Lucian Ciufudean wrote:
>>>>>> Sorry for all these iterations, here is a more consistent (but long)
>>>>>> email.
>>>>>>
>>>>> Iterations are fine as long as they bring more detail! Thanks for
>>>>> putting in all the work.
>>>>>
>>>>>> After not being able to run 'coverage run pyc_file.pyc', I created a
>>>>>> dummy driver main.py file that uses the compiled modules.
>>>>>>
>>>>>> root at spahire pyc]# ls -la
>>>>>> total 20
>>>>>> drwxr-xr-x. 2 root root 4096 Sep 18 13:20 .
>>>>>> drwxr-xr-x. 4 root root 4096 Sep 17 15:58 ..
>>>>>> -rw-r--r--. 1 root root 28 Sep 18 13:01 main.py
>>>>>> -rw-------. 1 root root 187 Sep 17 15:59 test_coverage_callee.pyc
>>>>>> -rw-------. 1 root root 212 Sep 17 15:59 test_coverage_caller.pyc
>>>>>> [root at spahire pyc]# cat main.py
>>>>>> import test_coverage_caller
>>>>>>
>>>>>> The source files for the 2 pyc files are here:
>>>>>>
>>>>>> [root at spahire pyc]# ls ../src
>>>>>> test_coverage_callee.py test_coverage_caller.py
>>>>>>
>>>>>> Further, I run coverage, I get a warning but a data file .coverage is
>>>>>> created:
>>>>>>
>>>>>> [root at spahire pyc]# coverage run --source=../src main.py
>>>>>> 1
>>>>>> 2
>>>>>> 2.1
>>>>>> 2.2
>>>>>> 2.3
>>>>>> Coverage.py warning: No data was collected.
>>>>>> [root at spahire pyc]# ls -l .coverage
>>>>>> -rw-r--r--. 1 root root 180 Sep 18 13:23 .coverage
>>>>>>
>>>>> The --source option tells coverage that the only files of interest are
>>>>> the ones in ../src. You never execute any files in ../src, so
>>>>> coverage
>>>>> hasn't collected any data.
>>>>>
>>>>>> --include does not make a difference:
>>>>>>
>>>>>> [root at spahire pyc]# coverage run --source=../src --include='*'
>>>>>> main.py
>>>>>> 1
>>>>>> 2
>>>>>> 2.1
>>>>>> 2.2
>>>>>> 2.3
>>>>>> Coverage.py warning: No data was collected.
>>>>>> [root at spahire pyc]# ls -l .coverage
>>>>>> -rw-r--r--. 1 root root 180 Sep 18 13:38 .coverage
>>>>>>
>>>>> Right, --source trumps --include. You've already told coverage that
>>>>> the
>>>>> only interesting files are in ../src, so there's nothing else to
>>>>> include. I should make a warning for inconsistent options like this
>>>>> (ticket:
>>>>> https://bitbucket.org/ned/coveragepy/issue/265/when-using-source-include-is-silently)
>>>>>
>>>>>> As expected, report shows nothing is covered:
>>>>>>
>>>>>> [root at spahire pyc]# coverage report --include='*test*'
>>>>>> Name Stmts Miss Cover
>>>>>> ----------------------------------------------------------------------------
>>>>>> /root/lucian/coverage/module1/src/test_coverage_callee 3 3
>>>>>> 0%
>>>>>> /root/lucian/coverage/module1/src/test_coverage_caller 3 3
>>>>>> 0%
>>>>>> ----------------------------------------------------------------------------
>>>>>> TOTAL 6 6 0%
>>>>>>
>>>>>> report does not accept --source, might this be the root to all evil?
>>>>>>
>>>>> Hmm, that seems like an oversight: I should make that possible
>>>>> (ticket:
>>>>> https://bitbucket.org/ned/coveragepy/issue/266/report-command-doesnt-accept-source-option)
>>>>>> [root at spahire pyc]# coverage report --include='*test*'
>>>>>> --source=../src
>>>>>> no such option: --source
>>>>>> Use 'coverage help' for help.
>>>>>>
>>>>>> No warning when using run without options, but the report is again
>>>>>> wrong:
>>>>>>
>>>>>> [root at spahire pyc]# coverage run main.py
>>>>>> 1
>>>>>> 2
>>>>>> 2.1
>>>>>> 2.2
>>>>>> 2.3
>>>>>> [root at spahire pyc]# coverage report
>>>>>> Name Stmts Miss Cover
>>>>>> ------------------------------------------
>>>>>> main 1 0 100%
>>>>>> test_coverage_callee NoSource: No source for code:
>>>>>> '/root/lucian/coverage/module1/pyc/test_coverage_callee.py'
>>>>>> test_coverage_caller NoSource: No source for code:
>>>>>> '/root/lucian/coverage/module1/pyc/test_coverage_caller.py'
>>>>>>
>>>>> The problem is that you haven't told coverage how to find the source
>>>>> files that correspond to your .pyc files. The --source option doesn't
>>>>> do that.
>>>>>
>>>>>> Same thing with a configuration file:
>>>>>> [root at spahire pyc]# cat .coveragerc
>>>>>> [run]
>>>>>> source=../src/
>>>>>> include=*
>>>>>>
>>>>> Right, same options, specified in a different way.
>>>>>> So now I turn to the api + the same .coveragerc, the same thing:
>>>>>>
>>>>>> [root at spahire pyc]# cat api.py
>>>>>> import coverage
>>>>>> cov = coverage.coverage()
>>>>>> cov.start()
>>>>>> import test_coverage_caller
>>>>>> cov.stop()
>>>>>> cov.save()
>>>>>>
>>>>>> [root at spahire pyc]# python api.py
>>>>>> 1
>>>>>> 2
>>>>>> 2.1
>>>>>> 2.2
>>>>>> 2.3
>>>>>> Coverage.py warning: No data was collected.
>>>>>>
>>>>> Another run with the same (non-)options.
>>>>>
>>>>>> As for your suggestion with [paths], the docs suggest to me that this
>>>>>> is for combining data.
>>>>> Yes, it is used when combining data, I should have fleshed out my idea
>>>>> more fully. Try creating a .coveragerc file like this:
>>>>>
>>>>> [run]
>>>>> parallel = true
>>>>>
>>>>> [paths]
>>>>> mysources =
>>>>> ../src
>>>>> .
>>>>>
>>>>> Then use "coverage run main.py", then "coverage combine", then
>>>>> "coverage
>>>>> report". If that works, we can talk about how to make it a bit
>>>>> easier.
>>>>>
>>>>> Hope that helps,
>>>>>
>>>>> --Ned.
>>>>>
>>>>>> Lucian
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 18, 2013 at 1:20 PM, Ned Batchelder
>>>>>> <ned at nedbatchelder.com
>>>>>> <mailto:ned at nedbatchelder.com>> wrote:
>>>>>>
>>>>>> On 9/18/13 3:42 AM, Lucian Ciufudean wrote:
>>>>>>> Here is how I ran coverage:
>>>>>>>
>>>>>>> coverage run --source=../src main.py
>>>>>>>
>>>>>>> main.py imports a pyc file that resides in the same folder.
>>>>>>> The
>>>>>>> source of this pyc file is available in ../src.
>>>>>>> (I created main.py just to go around the bug that you
>>>>>>> submitted
>>>>>>> above)
>>>>>>>
>>>>>> You don't mention the exact errors you saw (details matter),
>>>>>> but
>>>>>> in your previous message you mentioned the problem being while
>>>>>> reporting. You'll also have to specify the source directory
>>>>>> during the "coverage html" command (or coverage.html() call).
>>>>>> A
>>>>>> good way to do this is with a .coveragerc file.
>>>>>>
>>>>>> --Ned.
>>>>>>
>>>>>>
>>>>>>> Lucian
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 18, 2013 at 5:05 AM, Ned Batchelder
>>>>>>> <ned at nedbatchelder.com <mailto:ned at nedbatchelder.com>> wrote:
>>>>>>>
>>>>>>> On 9/17/13 7:42 AM, Lucian Ciufudean wrote:
>>>>>>>> Hi guys,
>>>>>>>>
>>>>>>>> I embarked on the road of obtaining a coverage report for
>>>>>>>> functional tests against a server process. The server is
>>>>>>>> deployed as .pyc files, and the source files can be made
>>>>>>>> available in a separate subversion working folder. I
>>>>>>>> wouldn't want to edit any existing .py files.
>>>>>>>>
>>>>>>>> Can coverage work with .pyc files - I am getting errors
>>>>>>>> when
>>>>>>>> using coverage run main.pyc from the command line, so
>>>>>>>> maybe
>>>>>>>> with the API?
>>>>>>> Hmm, you're right: "coverage run foo.pyc" does not work.
>>>>>>> I've
>>>>>>> created a ticket for this:
>>>>>>> https://bitbucket.org/ned/coveragepy/issue/264/coverage-wont-run-pyc-files
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> I tired this also, my custom code is run but it can not
>>>>>>>> find
>>>>>>>> the source files at the time of coverage.html_report()
>>>>>>>> although I passed the directory of source files to
>>>>>>>> coverage.coverage.
>>>>>>> You'll have to show details of how you tried to run
>>>>>>> coverage. If the .py files are in the places reported by
>>>>>>> your program, then it should work. You can also use the
>>>>>>> [paths] section of a .coveragerc to instruct coverage
>>>>>>> where
>>>>>>> the files are.
>>>>>>>
>>>>>>> --Ned.
>>>>>>>
>>>>>>>> Thanks a lot,
>>>>>>>> Lucian
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> testing-in-python mailing list
>>>>>>>> testing-in-python at lists.idyll.org
>>>>>>>> <mailto:testing-in-python at lists.idyll.org>
>>>>>>>> http://lists.idyll.org/listinfo/testing-in-python
>>>>> _______________________________________________
>>>>> testing-in-python mailing list
>>>>> testing-in-python at lists.idyll.org
>>>>> http://lists.idyll.org/listinfo/testing-in-python
>>>>>
>
More information about the testing-in-python
mailing list