[TIP] Resetting or Restarting coverage.py trace

Ned Batchelder ned at nedbatchelder.com
Tue Oct 18 03:33:47 PDT 2016


Just to be clear: there is a bit of hack code in coverage.py now, with
"WTW" as a moniker, but it is not a feature, it is a hack to test out
some theories about how a future feature might work.

--Ned.


On 10/17/16 11:31 AM, Shai Cantor wrote:
> Yup, that's what I'm doing right now.
> I've enabled the WTW feature and manually changing context upon a new
> test or time frame.
> The thing is, I cannot clear the contexts data, when I try to do that
> it never records anymore.
> I can only change context and keep the old.
>
> Any ideas?
>
> On Mon, Oct 17, 2016 at 6:25 PM,
> <testing-in-python-request at lists.idyll.org
> <mailto:testing-in-python-request at lists.idyll.org>> wrote:
>
>     Send testing-in-python mailing list submissions to
>             testing-in-python at lists.idyll.org
>     <mailto:testing-in-python at lists.idyll.org>
>
>     To subscribe or unsubscribe via the World Wide Web, visit
>             http://lists.idyll.org/listinfo/testing-in-python
>     <http://lists.idyll.org/listinfo/testing-in-python>
>     or, via email, send a message with subject or body 'help' to
>             testing-in-python-request at lists.idyll.org
>     <mailto:testing-in-python-request at lists.idyll.org>
>
>     You can reach the person managing the list at
>             testing-in-python-owner at lists.idyll.org
>     <mailto:testing-in-python-owner at lists.idyll.org>
>
>     When replying, please edit your Subject line so it is more specific
>     than "Re: Contents of testing-in-python digest..."
>
>     Today's Topics:
>
>        1. Re: Resetting or Restarting coverage.py trace (Ned Batchelder)
>
>
>     ---------- Forwarded message ----------
>     From: Ned Batchelder <ned at nedbatchelder.com
>     <mailto:ned at nedbatchelder.com>>
>     To: testing-in-python at lists.idyll.org
>     <mailto:testing-in-python at lists.idyll.org>
>     Cc: 
>     Date: Mon, 17 Oct 2016 11:25:08 -0400
>     Subject: Re: [TIP] Resetting or Restarting coverage.py trace
>
>     Another way would be to somehow get the "who tests what" feature
>     implemented, so that you could segregate your results by which
>     test covered the code. 
>     https://bitbucket.org/ned/coveragepy/issues/170/show-who-tests-what
>     <https://bitbucket.org/ned/coveragepy/issues/170/show-who-tests-what>
>
>     But that is a big effort.
>
>     --Ned.
>
>     On 10/17/16 11:17 AM, Shai Cantor wrote:
>>     I wish.
>>     In some setups, companies spin up servers that are dedicated for
>>     testing by multiple clients.
>>     Those cannot be frequently restarted.
>>     Do you think there can be another way?
>>
>>     On Mon, Oct 17, 2016 at 4:00 PM,
>>     <testing-in-python-request at lists.idyll.org
>>     <mailto:testing-in-python-request at lists.idyll.org>> wrote:
>>
>>         Send testing-in-python mailing list submissions to
>>                 testing-in-python at lists.idyll.org
>>         <mailto:testing-in-python at lists.idyll.org>
>>
>>         To subscribe or unsubscribe via the World Wide Web, visit
>>                 http://lists.idyll.org/listinfo/testing-in-python
>>         <http://lists.idyll.org/listinfo/testing-in-python>
>>         or, via email, send a message with subject or body 'help' to
>>                 testing-in-python-request at lists.idyll.org
>>         <mailto:testing-in-python-request at lists.idyll.org>
>>
>>         You can reach the person managing the list at
>>                 testing-in-python-owner at lists.idyll.org
>>         <mailto:testing-in-python-owner at lists.idyll.org>
>>
>>         When replying, please edit your Subject line so it is more
>>         specific
>>         than "Re: Contents of testing-in-python digest..."
>>
>>         Today's Topics:
>>
>>            1. Re: Resetting or Restarting coverage.py trace (Ned
>>         Batchelder)
>>            2. Re: pytest (James)
>>
>>
>>         ---------- Forwarded message ----------
>>         From: Ned Batchelder <ned at nedbatchelder.com
>>         <mailto:ned at nedbatchelder.com>>
>>         To: testing-in-python at lists.idyll.org
>>         <mailto:testing-in-python at lists.idyll.org>
>>         Cc: 
>>         Date: Mon, 17 Oct 2016 07:09:33 -0400
>>         Subject: Re: [TIP] Resetting or Restarting coverage.py trace
>>
>>         Wouldn't it be easiest to restart the process between the two
>>         phases?
>>
>>         --Ned.
>>
>>
>>         On 10/17/16 4:22 AM, Shai Cantor wrote:
>>>         I'm trying to find the coverage of multiple processes that
>>>         are running on multiple test phases.
>>>         For example:
>>>
>>>           * A CI server that is conducting tests that make HTTP
>>>             calls to another server for functional tests phase.
>>>           * The same CI server is conducting tests that make other
>>>             HTTP calls to the same server as part of the performance
>>>             test phase.
>>>
>>>         I would like to measure coverage for the functional test
>>>         phase, reset the coverage and start measuring again from
>>>         scratch for the performance test phase.
>>>
>>>         On Sun, Oct 16, 2016 at 10:00 PM,
>>>         <testing-in-python-request at lists.idyll.org
>>>         <mailto:testing-in-python-request at lists.idyll.org>> wrote:
>>>
>>>             Send testing-in-python mailing list submissions to
>>>                     testing-in-python at lists.idyll.org
>>>             <mailto:testing-in-python at lists.idyll.org>
>>>
>>>             To subscribe or unsubscribe via the World Wide Web, visit
>>>                    
>>>             http://lists.idyll.org/listinfo/testing-in-python
>>>             <http://lists.idyll.org/listinfo/testing-in-python>
>>>             or, via email, send a message with subject or body 'help' to
>>>                     testing-in-python-request at lists.idyll.org
>>>             <mailto:testing-in-python-request at lists.idyll.org>
>>>
>>>             You can reach the person managing the list at
>>>                     testing-in-python-owner at lists.idyll.org
>>>             <mailto:testing-in-python-owner at lists.idyll.org>
>>>
>>>             When replying, please edit your Subject line so it is
>>>             more specific
>>>             than "Re: Contents of testing-in-python digest..."
>>>
>>>             Today's Topics:
>>>
>>>                1. Resetting or Restarting coverage.py trace (Shai
>>>             Cantor)
>>>                2. Re: Resetting or Restarting coverage.py trace (Ned
>>>             Batchelder)
>>>                3. Re: are the absolute paths in .coverage necessary?
>>>             (Chris Withers)
>>>
>>>
>>>             ---------- Forwarded message ----------
>>>             From: Shai Cantor <shai at sealights.io
>>>             <mailto:shai at sealights.io>>
>>>             To: testing-in-python at lists.idyll.org
>>>             <mailto:testing-in-python at lists.idyll.org>
>>>             Cc: 
>>>             Date: Sun, 16 Oct 2016 09:36:42 +0300
>>>             Subject: [TIP] Resetting or Restarting coverage.py trace
>>>             Hi,
>>>
>>>             So I'm using coverage.py API
>>>
>>>             import coverage
>>>
>>>             cov = coverage.Coverage()
>>>             cov.start()
>>>
>>>             # .. call your code ..
>>>
>>>             cov.stop()
>>>             cov.save()
>>>
>>>
>>>             The main thread creates a coverage instance and starts it.
>>>             During the lifetime of the program I have multiple
>>>             threads and I need every once in while to reset or
>>>             restart the coverage process (or clear coverage data)
>>>             from one of the underlying threads.
>>>             Problem is that tracing is stopped after I do that. I
>>>             guess it's because I start it from another thread.
>>>
>>>             Is there a way I can reset/restart coverage from another
>>>             thread or manually clear coverage data?
>>>
>>>             Thanks
>>>
>>>
>>>
>>>             ---------- Forwarded message ----------
>>>             From: Ned Batchelder <ned at nedbatchelder.com
>>>             <mailto:ned at nedbatchelder.com>>
>>>             To: testing-in-python at lists.idyll.org
>>>             <mailto:testing-in-python at lists.idyll.org>
>>>             Cc: 
>>>             Date: Sun, 16 Oct 2016 12:10:48 -0400
>>>             Subject: Re: [TIP] Resetting or Restarting coverage.py trace
>>>             On 10/16/16 2:36 AM, Shai Cantor wrote:
>>>>             Hi,
>>>>
>>>>             So I'm using coverage.py API
>>>>             import coverage
>>>>
>>>>             cov = coverage.Coverage()
>>>>             cov.start()
>>>>
>>>>             # .. call your code ..
>>>>
>>>>             cov.stop()
>>>>             cov.save()
>>>>
>>>>             The main thread creates a coverage instance and starts it.
>>>>             During the lifetime of the program I have multiple
>>>>             threads and I need every once in while to reset or
>>>>             restart the coverage process (or clear coverage data)
>>>>             from one of the underlying threads.
>>>>             Problem is that tracing is stopped after I do that. I
>>>>             guess it's because I start it from another thread.
>>>>
>>>>             Is there a way I can reset/restart coverage from
>>>>             another thread or manually clear coverage data?
>>>>
>>>             I haven't heard this need before, to occasionally reset
>>>             or clear the coverage data from one of many threads. Can
>>>             you say more about what the bigger picture is here?
>>>             There might be an easier way to get it done.
>>>
>>>             --Ned.
>>>
>>>
>>>             ---------- Forwarded message ----------
>>>             From: Chris Withers <chris at simplistix.co.uk
>>>             <mailto:chris at simplistix.co.uk>>
>>>             To: Ned Batchelder <ned at nedbatchelder.com
>>>             <mailto:ned at nedbatchelder.com>>,
>>>             "testing-in-python at lists.idyll.org
>>>             <mailto:testing-in-python at lists.idyll.org>"
>>>             <testing-in-python at lists.idyll.org
>>>             <mailto:testing-in-python at lists.idyll.org>>
>>>             Cc: 
>>>             Date: Sun, 16 Oct 2016 19:44:38 +0100
>>>             Subject: Re: [TIP] are the absolute paths in .coverage
>>>             necessary?
>>>             Hi Ned,
>>>
>>>             Sorry, I meant: do the paths recorded in the .coverage
>>>             files need to be absolute or could Coverage be changed
>>>             to just record the path relatively to the current
>>>             working directory at the start of execution?
>>>
>>>             cheers,
>>>
>>>             Chris
>>>
>>>
>>>             On 12/10/2016 22:31, Ned Batchelder wrote:
>>>
>>>                 Chris, as far as I can tell, they do not have to be
>>>                 absolute. As you can
>>>                 see from the sample of my rc file, you can replace
>>>                 an arbitrary prefix
>>>                 with *, and it will work just fine.  How about this:
>>>
>>>                 [paths]
>>>                 mylib =
>>>                    mylib
>>>                    */workspace/PYTHON/*/mylib
>>>                 tests =
>>>                    tests
>>>                    */workspace/PYTHON/*/tests
>>>
>>>                 --Ned.
>>>
>>>
>>>                 On 10/12/16 10:59 AM, Chris Withers wrote:
>>>
>>>                     What I currently have is:
>>>
>>>                     [paths]
>>>                     mylib =
>>>                        mylib
>>>                      
>>>                      /data/jenkins/jobs/mylib-test/workspace/PYTHON/*/mylib
>>>                     tests =
>>>                        tests
>>>                      
>>>                      /data/jenkins/jobs/mylib-test/workspace/PYTHON/*/tests
>>>
>>>                     How can I remove the absolute bits of that?
>>>
>>>                     But, back to my original question: why are the
>>>                     paths in .coverage
>>>                     absolute in the first place? :-)
>>>
>>>                     Chris
>>>
>>>                     On 12/10/2016 15:53, Ned Batchelder wrote:
>>>
>>>                         Chris, can you show the absolute paths you
>>>                         are using?  I have this in my
>>>                         rc file to deal with varieties of CI:
>>>
>>>                              [paths]
>>>                              source =
>>>                                  .
>>>                                  *\coverage\trunk
>>>                                  */coverage/trunk
>>>                                  *\coveragepy
>>>
>>>                         --Ned.
>>>
>>>
>>>                         On 10/12/16 10:00 AM, Chris Withers wrote:
>>>
>>>                             Hi Ned,
>>>
>>>                             Are the absolute paths necessary in
>>>                             .coverage?
>>>
>>>                             The reason I ask is that if, for
>>>                             example, they were relative to the
>>>                             cwd, then I wouldn't need a .coveragerc
>>>                             with a whole bunch of paths
>>>                             predicated on the exact filesystem
>>>                             layout of my CI server.
>>>
>>>                             thoughts?
>>>
>>>                             Chris
>>>
>>>
>>>
>>>
>>>
>>>             _______________________________________________
>>>             testing-in-python mailing list
>>>             testing-in-python at lists.idyll.org
>>>             <mailto:testing-in-python at lists.idyll.org>
>>>             http://lists.idyll.org/listinfo/testing-in-python
>>>             <http://lists.idyll.org/listinfo/testing-in-python>
>>>
>>>
>>>
>>>
>>>         _______________________________________________
>>>         testing-in-python mailing list
>>>         testing-in-python at lists.idyll.org
>>>         <mailto:testing-in-python at lists.idyll.org>
>>>         http://lists.idyll.org/listinfo/testing-in-python
>>>         <http://lists.idyll.org/listinfo/testing-in-python>
>>         ---------- Forwarded message ---------- From: James
>>         <bjlockie at lockie.ca <mailto:bjlockie at lockie.ca>> To: Bruno
>>         Oliveira <nicoddemus at gmail.com
>>         <mailto:nicoddemus at gmail.com>>,
>>         testing-in-python at lists.idyll.org
>>         <mailto:testing-in-python at lists.idyll.org> Cc:  Date: Mon, 17
>>         Oct 2016 09:00:39 -0400 Subject: Re: [TIP] pytest
>>
>>         The output was: a = <testscript.TestAclasss instance at
>>         0x7f3ca6bf83b0>     def test_foo(a): >       assert a == '1'
>>         E       assert <testscript.TestAclasss instance at
>>         0x7f3ca6bf83b0> == '1' testscript.py:6: AssertionError 1
>>         failed in 0.03 seconds
>>
>>         The missing "self" fixed it.
>>
>>         On 2016-10-16 12:56 AM, Bruno Oliveira wrote:
>>>
>>>         Hi James,
>>>
>>>         What error do you get? From your example it is missing a
>>>         |self| argument, otherwise it should work just fine.
>>>
>>>         Cheers, Bruno.
>>>
>>>>>>         On Sat, Oct 15, 2016 at 11:03 AM James <bjlockie at lockie.ca
>>>         <mailto:bjlockie at lockie.ca>> wrote:
>>>
>>>             I tried to make it work with a class:
>>>
>>>             import pytest # testscript.py class TestAclasss:
>>>
>>>                 def test_foo(a):         assert a == '1'
>>>
>>>             Do I need to do something in the __init__ function?
>>>             On 2016-10-14 02:50 PM, Bruno Oliveira wrote:
>>>>
>>>>             Hi James,
>>>>
>>>>             Thanks for posting the complete example.
>>>>
>>>>             There are some problems here, both general Python
>>>>             problems and some related to how you are using pytest:
>>>>
>>>>              1. |testscript.py| has no access to the |A| function
>>>>                 because it doesn’t import it.
>>>>              2. You probably want to declare your fixture in a
>>>>                 |conftest.py| file so pytest can make it available
>>>>                 for all tests in your test suite.
>>>>              3. You have to first declare your option using the
>>>>                 |pytest_addoption| hook.
>>>>              4. Since you intend to provide the value passed in the
>>>>                 command line to the tests using a fixture, it is
>>>>                 simpler to skip the test in the fixture itself.
>>>>
>>>>             Here’s the full example with my suggestions:
>>>>
>>>>             |# conftest.py import pytest
>>>>             @pytest.fixture(scope='module') def a(request): value =
>>>>             request.config.getoption("A") if not value:
>>>>             pytest.skip('test needs -A option to run') return value
>>>>             def pytest_addoption(parser): parser.addoption("--A",
>>>>             action="store", default=None, help="a option") #
>>>>             testscript.py def test_foo(a): assert a == '1' |
>>>>
>>>>             Running it:
>>>>
>>>>             |$ py.test testscript.py -q s 1 skipped in 0.01 seconds
>>>>             $ py.test testscript.py -q --A 1 . 1 passed in 0.01
>>>>             seconds $ py.test testscript.py -q --A 2 F
>>>>             ================================== FAILURES
>>>>             ===================================
>>>>             __________________________________ test_foo
>>>>             ___________________________________ a = '2' def
>>>>             test_foo(a): > assert a == '1' E assert '2' == '1' E -
>>>>             2 E + 1 testscript.py:6: AssertionError 1 failed in
>>>>             0.07 seconds |
>>>>
>>>>             Hope that helps, Bruno.
>>>>
>>>>>>>>             On Fri, Oct 14, 2016 at 3:07 PM James
>>>>             <bjlockie at lockie.ca <mailto:bjlockie at lockie.ca>> wrote:
>>>>
>>>>                 Here is the exact code: $ cat testconfig.py import
>>>>                 pytest
>>>>                 @pytest.fixture(scope='module') def A(request):    
>>>>                 return request.config.getoption("--A")
>>>>                 $ cat testscript.py import pytest
>>>>                   skip_A = pytest.mark.skipif(    
>>>>                 A(pytest.config.request),     reason="need --A
>>>>                 option to run" )
>>>>                 $ py.test testscript.py
>>>>                 ============================= test session starts
>>>>                 ============================== platform linux2 --
>>>>                 Python 2.7.12, pytest-2.8.7, py-1.4.31,
>>>>                 pluggy-0.3.1 rootdir: /home/rjl/pytest, inifile:
>>>>                 collected 0 items / 1 errors
>>>>                 ==================================== ERRORS
>>>>                 ====================================
>>>>                 ________________________ ERROR collecting
>>>>                 testscript.py ________________________
>>>>                 testscript.py:4: in <module>    
>>>>                 A(pytest.config.request),
>>>>                 E   NameError: name 'A' is not defined
>>>>                 =========================== 1 error in 0.01 seconds
>>>>                 ============================ Instead of doing
>>>>                 pytest.config.getoptionmore than one like this:
>>>>                 skip_A = pytest.mark.skipif(
>>>>                 pytest.config.getoption("--A"), reason="need --A
>>>>                 option to run" ) I want to do it once in the config
>>>>                 file. I might want to use the value of the argument
>>>>                 --A elsewhere so I don't want to do getoption many
>>>>                 times.
>>>>                 On 2016-10-14 01:16 PM, Bruno Oliveira wrote:
>>>>>
>>>>>                 Hi,
>>>>>
>>>>>                 After fixing the missing “pytest” import, I get
>>>>>                 this error:
>>>>>
>>>>>                 |foo.py:8: in <module> A(pytest.config.request), E
>>>>>                 AttributeError: 'Config' object has no attribute
>>>>>                 'request' |
>>>>>
>>>>>                 Please make sure to post a complete example and
>>>>>                 state clearly what your problem is and what you
>>>>>                 are trying to accomplish.
>>>>>
>>>>>                 Cheers, Bruno.
>>>>>
>>>>>>>>>>                 On Fri, Oct 14, 2016 at 2:03 PM James
>>>>>                 <bjlockie at lockie.ca <mailto:bjlockie at lockie.ca>>
>>>>>                 wrote:
>>>>>
>>>>>                     I have this in my config file:
>>>>>                     @pytest.fixture(scope='module') def
>>>>>                     A(request):     return
>>>>>                     request.config.getoption("--A")   skip_A =
>>>>>                     pytest.mark.skipif(    
>>>>>                     A(pytest.config.request),     reason="need --A
>>>>>                     option to run" ) but when I run it, it says
>>>>>                     E   NameError: name 'A' is not defined  
>>>>>                     _______________________________________________
>>>>>                     testing-in-python mailing list
>>>>>                     testing-in-python at lists.idyll.org
>>>>>                     <mailto:testing-in-python at lists.idyll.org>
>>>>>                     http://lists.idyll.org/listinfo/testing-in-python
>>>>>                     <http://lists.idyll.org/listinfo/testing-in-python>
>>>>>
>>>>>
>>         _______________________________________________
>>         testing-in-python mailing list
>>         testing-in-python at lists.idyll.org
>>         <mailto:testing-in-python at lists.idyll.org>
>>         http://lists.idyll.org/listinfo/testing-in-python
>>         <http://lists.idyll.org/listinfo/testing-in-python> 
>>
>>     _______________________________________________
>>     testing-in-python mailing list
>>     testing-in-python at lists.idyll.org
>>     <mailto:testing-in-python at lists.idyll.org>
>>     http://lists.idyll.org/listinfo/testing-in-python
>>     <http://lists.idyll.org/listinfo/testing-in-python>
>     _______________________________________________ testing-in-python
>     mailing list testing-in-python at lists.idyll.org
>     <mailto:testing-in-python at lists.idyll.org>
>     http://lists.idyll.org/listinfo/testing-in-python
>     <http://lists.idyll.org/listinfo/testing-in-python> 
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20161018/3503919b/attachment-0001.html>


More information about the testing-in-python mailing list