[TIP] Bug with nosetests? Adding a new case with autospec=True will take 15 seconds

Michael Foord fuzzyman at voidspace.org.uk
Sun Mar 10 12:53:12 PDT 2013 

On 10 Mar 2013, at 00:34, John Wong <gokoproject at gmail.com> wrote:

> Hi,
> Sorry for the spam. Actually, I looked at it again. I feel like the way I was patching was fundamentally incorrect. 
> 
> #bitbucket/apis.py
> import requests
> 
> def foo(..):
>     requests.get(...)
> 
> then in my test I should just patch.object(requests, 'get') and not patch('bitbucket.apis.requests', autospec=True)
> 
> I think that's one mistake I made, if I read the doc correctly. 


Both are legitimate. When using autospec patching requests.get(...) is probably more *efficient*. Have you tried timing it and does it help?

Have you confirmed yet if the one test class does take all the extra time when run on its own?

Michael


> Thanks.
> 
> John
> On Sat, Mar 9, 2013 at 7:12 PM, John Wong <gokoproject at gmail.com> wrote:
> Thanks! Finally can get back to this.
> Yes. I ahve to agree that I should just patch out certain things like requests.get, and requests.post only.
> 
> (bbpy)yeukhon at yeukhon-P5E-VM-DO:~/hg/bitbucket-python-api/tests/small$ nosetests test_bitbucket.py
> ...
> ----------------------------------------------------------------------
> Ran 3 tests in 0.103s
> 
> OK
> 
> 
> This is an individual test. It ran 3 tests. One of which is a new test class I tried to add (which just has one test case)
> 
> If I comment out the new test class in test_bitbucket.py, and ran the entire suite:
> (bbpy)yeukhon at yeukhon-P5E-VM-DO:~/hg/bitbucket-python-api/tests/small$ nosetests
> .................
> ----------------------------------------------------------------------
> Ran 17 tests in 1.020s
> 
> OK
> 
> As you can see. For some strange reasons, a new test class would trigger a big jump!
> 
> I did a little debugging using nose's set_tace()
> 
> run "nosetest" with debug: http://pastebin.com/YK7yXrtG
> run "nsoetest test_bitbucket.py" http://pastebin.com/WbfEYKnL
> 
> I can understand the one with all tests would do some discovery so the debug log looks longer. 
> 
> John
> 
> On Fri, Mar 8, 2013 at 8:40 AM, Michael Foord <fuzzyman at voidspace.org.uk> wrote:
> 
> On 7 Mar 2013, at 19:52, Dan Wandschneider <daniel.wandschneider at schrodinger.com> wrote:
> 
> > I have some guesses about this, which I elaborated on in:
> > http://stackoverflow.com/a/15280350/1309332
> >
> > But basically, it looks like the recursive calls to create_autospec() are slowing down on your mocked object (whatever it is).  ID recommend one of three approaches:
> > 1. Don't autospec.  Do you need it?
> > 2. Only autospec some important method within the complex object that you are replacing
> > 3. use instance=True to limit the depth of recursion of calls to create_autospec().  This may change behavior that you are depending upon, however.  Be careful, since you are essentially lying to mock.patch if you do this.
> >
> 
> 
> The issue is that adding one test (class/module ?)  slows the test run down by 13 seconds or so. What is needed is running that test on its own (with create_autospec on) to see if it actually takes that long or whether there is some additional mysterious source of slowdown (which the OP seems to suspect).
> 
> Adding "instance=True" to *the patch call* is probably a bad idea.
> 
> It is entirely *possible* that autospec is really, really slow in some cases - but to be that much slower is very odd.
> 
> Michael
> 
> > Good luck!
> > -Dan W.
> >
> >
> >
> > On Wed, Mar 6, 2013 at 12:00 PM, <testing-in-python-request at lists.idyll.org> wrote:
> > Send testing-in-python mailing list submissions to
> >         testing-in-python at lists.idyll.org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         http://lists.idyll.org/listinfo/testing-in-python
> > or, via email, send a message with subject or body 'help' to
> >         testing-in-python-request at lists.idyll.org
> >
> > You can reach the person managing the list at
> >         testing-in-python-owner at lists.idyll.org
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of testing-in-python digest..."
> >
> > Today's Topics:
> >
> >    1. Bug with nosetests? Adding a new case with autospec=True will
> >       take 15 seconds (John Wong)
> >
> >
> > ---------- Forwarded message ----------
> > From: John Wong <gokoproject at gmail.com>
> > To: testing-in-python at lists.idyll.org
> > Cc:
> > Date: Tue, 5 Mar 2013 22:57:29 -0500
> > Subject: [TIP] Bug with nosetests? Adding a new case with autospec=True will take 15 seconds
> > HI,
> >
> > Thanks for all the help so far.
> >
> > Here is the repo: https://bitbucket.org/yeukhon/bitbucket-python-api/src/6e57e95d5d09/tests/small?at=default
> >
> > I have a base unit test class in base.py.  For test_bitbucket.py, I inherit from the base unit test class.
> > I want to add new test cases and classes to test_bitbucket.py While doing so, I discovered performance issue.
> >
> > I ran `nosetests` inside tests/small and I am getting these mix results
> >
> >
> > Ran 18 tests in 14.523s        - autospec=True and test_account_creation exists
> >
> > Ran 18 tests in 0.621s         - autospec=False and test_account_creation_exists
> >
> >
> > Ran 17 tests in 1.081s         - autospec=True and test_account_creation is deleted
> >
> > Ran 17 tests in 0.090s         - autospec=False, and test_account_creation is deleted
> >
> >
> > I understand requests is a big library, but the performance hit is crazy, going from 1.1s to 14.5s with and without the new test class, respectively.
> >
> > Interestingly, if we run test_bitbucket.py individually.
> >
> > (bbpy)yeukhon at yeukhon-P5E-VM-DO:~/hg/bitbucket-python-api/tests/small$ nosetests test_bitbucket.py
> > ...
> > ----------------------------------------------------------------------
> > Ran 3 tests in 0.090s
> >
> > OK
> >
> >
> > Pretty much the same with/without autospec on.
> >
> > Any idea? I don't know enough about nosetests internal.
> >
> > Thanks!
> >
> > John
> >
> > _______________________________________________
> > testing-in-python mailing list
> > testing-in-python at lists.idyll.org
> > http://lists.idyll.org/listinfo/testing-in-python
> >
> >
> > _______________________________________________
> > testing-in-python mailing list
> > testing-in-python at lists.idyll.org
> > http://lists.idyll.org/listinfo/testing-in-python
> 
> 
> --
> http://www.voidspace.org.uk/
> 
> 
> May you do good and not evil
> May you find forgiveness for yourself and forgive others
> May you share freely, never taking more than you give.
> -- the sqlite blessing
> http://www.sqlite.org/different.html
> 
> 
> 
> 
> 
> 
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
> 
> 


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html

More information about the testing-in-python mailing list