Firstly, I would like to say that TiP BoF was amazing. I was surprised about how many people went to it, and it's awesome to see people "passionate about testing" (saying with an accent). <div><br></div><div>
Nevertheless, one thing I definitely learned from TiP BoF and Gary Bernhardt's Talk on "Units Needs Testing Too" (<a href="http://pycon.blip.tv/file/4879997/">http://pycon.blip.tv/file/4879997/</a>) is that if you want fast unit tests, you need to write tests that cover localized code to get the quicker feedback. </div>
<div><br></div><div>This means reducing the number of integration points or mocking out where an exhaustive test simply isn't necessary for the case. With Django, there seems to be a tendency to exhaustively test views. This creates a scenario where your test integrates with context processors, middleware, database filters and probably a number of other items you may be unaware of. In short, if your test case isn't about integration or exhaustive acceptance, then mock out those areas that aren't required.</div>
<div><br></div><div>At my company we're working on moving towards a more localized unit test suite, while still maintaining our integration test suite. Running the localized unit test suite allows us to run hundreds of tests in a matter of a second whereas our integration and automation tests will take longer, but won't run as often.</div>
<div><br></div><div>That's all I can say, but I'm sure glad to be apart of this new community I've discovered!</div><div><br></div><div>- Thomas</div><div><br><div class="gmail_quote">On Tue, Mar 15, 2011 at 3:00 PM, <span dir="ltr"><<a href="mailto:testing-in-python-request@lists.idyll.org">testing-in-python-request@lists.idyll.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Send testing-in-python mailing list submissions to<br>
<a href="mailto:testing-in-python@lists.idyll.org">testing-in-python@lists.idyll.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.idyll.org/listinfo/testing-in-python" target="_blank">http://lists.idyll.org/listinfo/testing-in-python</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:testing-in-python-request@lists.idyll.org">testing-in-python-request@lists.idyll.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:testing-in-python-owner@lists.idyll.org">testing-in-python-owner@lists.idyll.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of testing-in-python digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. TiP BoF feedback & call for help (Jorge Vargas)<br>
2. Re: TiP BoF feedback & call for help (Alfredo Deza)<br>
3. Re: TiP BoF feedback & call for help (Mark Sienkiewicz)<br>
4. Re: TiP BoF feedback & call for help (Greg Turnquist)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Mon, 14 Mar 2011 15:11:11 -0400<br>
From: Jorge Vargas <<a href="mailto:jorge.vargas@gmail.com">jorge.vargas@gmail.com</a>><br>
Subject: [TIP] TiP BoF feedback & call for help<br>
To: TesttingInPython <<a href="mailto:testing-in-python@lists.idyll.org">testing-in-python@lists.idyll.org</a>><br>
Message-ID:<br>
<AANLkTi=VfnYuc-w+-0boaby0hchgr2GXM=<a href="mailto:86ZhXni2DA@mail.gmail.com">86ZhXni2DA@mail.gmail.com</a>><br>
Content-Type: text/plain; charset=ISO-8859-1<br>
<br>
Hello all,<br>
<br>
First of all although I dont' think there is anything with a different<br>
opinion I totally loved it.<br>
<br>
It's a shame I missed it last year.<br>
<br>
Second (and I'm sorry if you did this but I got in a little late) I<br>
thought it will be more like other BoF where people will discuss<br>
rather than lighting talks. Because I was hoping to get some help to<br>
find out why our tests are so slow. So here is my lightning talk<br>
regarding that.<br>
<br>
Hi, I got a problem ( halp! )<br>
<picture of a sad kitty><br>
My tests run slooooooooow<br>
But I don't know who to blame<br>
This is django so it's probably it's fault.<br>
<haters got to hate><br>
Or perhaps it's that my boss does not want to let me move them over to<br>
nose Still stuck on vanilla unittest :(<br>
Or perhaps it's just a bad setup ? more fixtures around ?<br>
Or maybe it's some of the crazyness we do to have a standalone package<br>
that does not require a django project.<br>
We currently have 501 tests on our model only.<br>
ALL of them just check our business logic and raise validation errors<br>
for different problems and such.<br>
Running under jenkins in a VPS with sqlite and postgres (so 1002 runs)<br>
it takes anywhere from 20-30minutes to run.<br>
<br>
So who is wrong? me? us? jenkins? django? unittest ? Who to blame ?<br>
I'll love to run my tests more but 30 minutes feedback is just crazy.<br>
<br>
Looking forward for Santa Clara !<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Mon, 14 Mar 2011 15:26:19 -0400<br>
From: Alfredo Deza <<a href="mailto:alfredodeza@gmail.com">alfredodeza@gmail.com</a>><br>
Subject: Re: [TIP] TiP BoF feedback & call for help<br>
To: Jorge Vargas <<a href="mailto:jorge.vargas@gmail.com">jorge.vargas@gmail.com</a>><br>
Cc: TesttingInPython <<a href="mailto:testing-in-python@lists.idyll.org">testing-in-python@lists.idyll.org</a>><br>
Message-ID:<br>
<<a href="mailto:AANLkTikq9sr%2B1r0B1Gfx7w7E7Otox8rP8LC1sYeJt4E7@mail.gmail.com">AANLkTikq9sr+1r0B1Gfx7w7E7Otox8rP8LC1sYeJt4E7@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
On Mon, Mar 14, 2011 at 3:11 PM, Jorge Vargas <<a href="mailto:jorge.vargas@gmail.com">jorge.vargas@gmail.com</a>>wrote:<br>
<br>
> Hello all,<br>
><br>
> First of all although I dont' think there is anything with a different<br>
> opinion I totally loved it.<br>
><br>
> It's a shame I missed it last year.<br>
><br>
> Second (and I'm sorry if you did this but I got in a little late) I<br>
> thought it will be more like other BoF where people will discuss<br>
> rather than lighting talks. Because I was hoping to get some help to<br>
> find out why our tests are so slow. So here is my lightning talk<br>
> regarding that.<br>
><br>
> Hi, I got a problem ( halp! )<br>
> <picture of a sad kitty><br>
> My tests run slooooooooow<br>
> But I don't know who to blame<br>
> This is django so it's probably it's fault.<br>
> <haters got to hate><br>
> Or perhaps it's that my boss does not want to let me move them over to<br>
> nose Still stuck on vanilla unittest :(<br>
> Or perhaps it's just a bad setup ? more fixtures around ?<br>
> Or maybe it's some of the crazyness we do to have a standalone package<br>
> that does not require a django project.<br>
> We currently have 501 tests on our model only.<br>
> ALL of them just check our business logic and raise validation errors<br>
> for different problems and such.<br>
> Running under jenkins in a VPS with sqlite and postgres (so 1002 runs)<br>
> it takes anywhere from 20-30minutes to run.<br>
><br>
> So who is wrong? me? us? jenkins? django? unittest ? Who to blame ?<br>
> I'll love to run my tests more but 30 minutes feedback is just crazy.<br>
><br>
<br>
Independently of why they are slow, I would play with py.test and the xdist<br>
plugin that will<br>
allow you to run tests in parallel.<br>
<br>
At work, we went from 20 minute test runs to around 5.<br>
<br>
Since your tests are "vanilla unittest" you should be able to try out<br>
py.test without issues.<br>
<br>
For best results, try to match the number of processes to the number of<br>
cores on your machine.<br>
<br>
On our 4 core testing server, the command to run in parallel looks like<br>
this:<br>
<br>
py.test -n 4<br>
<br>
It should work out of the box :)<br>
<br>
<br>
You need to install the following:<br>
<br>
pip install pytest<br>
pip install pytest-xdist<br>
<br>
<br>
In that order. Please post back if you do try it to see how it went!<br>
<br>
><br>
> Looking forward for Santa Clara !<br>
><br>
> _______________________________________________<br>
> testing-in-python mailing list<br>
> <a href="mailto:testing-in-python@lists.idyll.org">testing-in-python@lists.idyll.org</a><br>
> <a href="http://lists.idyll.org/listinfo/testing-in-python" target="_blank">http://lists.idyll.org/listinfo/testing-in-python</a><br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.idyll.org/pipermail/testing-in-python/attachments/20110314/b3ae49e3/attachment-0001.htm" target="_blank">http://lists.idyll.org/pipermail/testing-in-python/attachments/20110314/b3ae49e3/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Mon, 14 Mar 2011 18:05:29 -0400<br>
From: Mark Sienkiewicz <<a href="mailto:sienkiew@stsci.edu">sienkiew@stsci.edu</a>><br>
Subject: Re: [TIP] TiP BoF feedback & call for help<br>
To: Jorge Vargas <<a href="mailto:jorge.vargas@gmail.com">jorge.vargas@gmail.com</a>><br>
Cc: TesttingInPython <<a href="mailto:testing-in-python@lists.idyll.org">testing-in-python@lists.idyll.org</a>><br>
Message-ID: <<a href="mailto:4D7E9129.30306@stsci.edu">4D7E9129.30306@stsci.edu</a>><br>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br>
<br>
<br>
> So who is wrong? me? us? jenkins? django? unittest ? Who to blame ?<br>
> I'll love to run my tests more but 30 minutes feedback is just crazy.<br>
><br>
<br>
You'll just have to start measuring things. Blaming django doesn't get<br>
you anywhere even if it is accurate -- you have to understand what is<br>
happening in your system so you can know what you can do different.<br>
<br>
First, how do you know there is anything wrong? Your average throughput<br>
works out to about 1.8 seconds per test. Is it possible that 20-30<br>
minutes is actually _fast_ for what you are doing? If not, how do you know?<br>
<br>
Are all the tests taking 1.8 seconds +/- 2% ? Or do 90% of your tests<br>
take 0.1 seconds each and 10% of your tests take 17 seconds each? What<br>
do the fast ones have in common? What do the slow ones have in common?<br>
<br>
If you run a single test, how long does it take? If you run 10 tests,<br>
does it take 10 times as long? Do 100 tests take 100 times as long?<br>
<br>
If you're using unittest, how long does it spend in the test method, how<br>
long in the class setup, how long in the overhead code?<br>
<br>
Do your tests log in to the postgres server 500 times, or only once?<br>
And are the postgres tests faster or slower than the same tests run on<br>
sqlite?<br>
<br>
Do you run all your tests in a single process? If not, how many<br>
processes and how much process creation time? (I ran truss on my python<br>
interpreter once and saw about 3000 file open calls just to start an<br>
interactive interpreter.)<br>
<br>
Once you collect some data (maybe even just a few minutes - you might<br>
not need a whole test run), you have some clues. There is no point in<br>
optimizing until you know what needs to be better. If you just poke<br>
about at random, you might completely optimize something that is<br>
responsible for 1% of your run time -- but you don't care if you save 18<br>
seconds out of half an hour.<br>
<br>
Mark<br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Mon, 14 Mar 2011 17:15:38 -0500<br>
From: Greg Turnquist <<a href="mailto:greg.l.turnquist@gmail.com">greg.l.turnquist@gmail.com</a>><br>
Subject: Re: [TIP] TiP BoF feedback & call for help<br>
To: TesttingInPython <<a href="mailto:testing-in-python@lists.idyll.org">testing-in-python@lists.idyll.org</a>><br>
Message-ID:<br>
<AANLkTimg6hsBCubDg8GQpFAYr=+<a href="mailto:zjg9cmUscwF3qEp07@mail.gmail.com">zjg9cmUscwF3qEp07@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
As they say, premature optimization is the root of all evil. Profile and<br>
determine where all your time is being spent.<br>
<br>
On Mon, Mar 14, 2011 at 5:05 PM, Mark Sienkiewicz <<a href="mailto:sienkiew@stsci.edu">sienkiew@stsci.edu</a>>wrote:<br>
<br>
><br>
> So who is wrong? me? us? jenkins? django? unittest ? Who to blame ?<br>
>> I'll love to run my tests more but 30 minutes feedback is just crazy.<br>
>><br>
>><br>
><br>
> You'll just have to start measuring things. Blaming django doesn't get you<br>
> anywhere even if it is accurate -- you have to understand what is happening<br>
> in your system so you can know what you can do different.<br>
><br>
> First, how do you know there is anything wrong? Your average throughput<br>
> works out to about 1.8 seconds per test. Is it possible that 20-30 minutes<br>
> is actually _fast_ for what you are doing? If not, how do you know?<br>
><br>
> Are all the tests taking 1.8 seconds +/- 2% ? Or do 90% of your tests take<br>
> 0.1 seconds each and 10% of your tests take 17 seconds each? What do the<br>
> fast ones have in common? What do the slow ones have in common?<br>
><br>
> If you run a single test, how long does it take? If you run 10 tests, does<br>
> it take 10 times as long? Do 100 tests take 100 times as long?<br>
><br>
> If you're using unittest, how long does it spend in the test method, how<br>
> long in the class setup, how long in the overhead code?<br>
><br>
> Do your tests log in to the postgres server 500 times, or only once? And<br>
> are the postgres tests faster or slower than the same tests run on sqlite?<br>
><br>
> Do you run all your tests in a single process? If not, how many processes<br>
> and how much process creation time? (I ran truss on my python interpreter<br>
> once and saw about 3000 file open calls just to start an interactive<br>
> interpreter.)<br>
><br>
> Once you collect some data (maybe even just a few minutes - you might not<br>
> need a whole test run), you have some clues. There is no point in<br>
> optimizing until you know what needs to be better. If you just poke about<br>
> at random, you might completely optimize something that is responsible for<br>
> 1% of your run time -- but you don't care if you save 18 seconds out of half<br>
> an hour.<br>
><br>
> Mark<br>
><br>
><br>
><br>
> _______________________________________________<br>
> testing-in-python mailing list<br>
> <a href="mailto:testing-in-python@lists.idyll.org">testing-in-python@lists.idyll.org</a><br>
> <a href="http://lists.idyll.org/listinfo/testing-in-python" target="_blank">http://lists.idyll.org/listinfo/testing-in-python</a><br>
><br>
<br>
<br>
<br>
--<br>
Greg Turnquist (<a href="mailto:Greg.L.Turnquist@gmail.com">Greg.L.Turnquist@gmail.com</a>)<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.idyll.org/pipermail/testing-in-python/attachments/20110314/d0b8b606/attachment-0001.htm" target="_blank">http://lists.idyll.org/pipermail/testing-in-python/attachments/20110314/d0b8b606/attachment-0001.htm</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
testing-in-python mailing list<br>
<a href="mailto:testing-in-python@lists.idyll.org">testing-in-python@lists.idyll.org</a><br>
<a href="http://lists.idyll.org/listinfo/testing-in-python" target="_blank">http://lists.idyll.org/listinfo/testing-in-python</a><br>
<br>
<br>
End of testing-in-python Digest, Vol 50, Issue 10<br>
*************************************************<br>
</blockquote></div><br></div>