[TIP] why do you use py.test?

[meme dough]

Hi,

I think all the test runners are good, though I would like to point
out the the speed comparison here isn't right.

There is only a small sub second difference due to the once off start
up time - and it's sub second.  Whilst humans do have time perception
in the sub second range, really this is still so small that it won't
prevent or slow down TDD.  I do TDD with pytest all the time and I use
nosier (disclaimer I wrote it) to automatically run all tests whenever
a file is changed.

Running 100 very fast tests all like:

def test_1():
    assert 1 == 1

pytest:
platform linux2 -- Python 2.6.5 -- pytest-1.3.4
test path 1: tests/test_foo.py
tests/test_foo.py
....................................................................................................
real	0m0.481s
user	0m0.430s
sys	0m0.040s


nose:
....................................................................................................
----------------------------------------------------------------------
Ran 100 tests in 0.009s
OK
real	0m0.151s
user	0m0.110s
sys	0m0.040s


Running 100 slow tests all like:

def test_1():
    time.sleep(1)


pytest:
platform linux2 -- Python 2.6.5 -- pytest-1.3.4
test path 1: tests/test_foo.py
tests/test_foo.py
....................................................................................................
real	1m40.596s
user	0m0.460s
sys	0m0.060s


nose:
....................................................................................................
----------------------------------------------------------------------
Ran 100 tests in 100.103s
OK
real	1m40.248s
user	0m0.130s
sys	0m0.030s


So in both cases (fast tests and slow tests) the difference is in the
0.3-0.4 sec range.

Both are more than usable (as I'm sure unitest / unittest2 are as
well) for TDD and getting though fast tests fast.

pytest has a big advantage with distributed testing to make use of
multiple cores and multiple hosts which makes a big difference if the
test suite grows.  This will allow you to scale the CI system to keep
tests running fast even with a very large test suite.  There is also a
coverage plugin (disclaimer I wrote it) that works with distributed
testing over multiple cores / hosts and also subprocesses.

nose can run in parallel, however it is limited to multiple cores on
the 1 host.  Also the multiprocess plugin doesn't play well with other
plugins (specifically cover).  I know someone made the cover plugin
work with the multiprocess plugin at some point, does anyone know if
that made it back into nose 1.0?

:)

On 19 February 2011 13:44, Herman Sheremetyev <herman at swebpage.com> wrote:
> Thanks for the explanations, everyone! I think I have a better grasp
> on this now :)
>
> (some comments inline)
>
> On Tue, Feb 15, 2011 at 7:12 PM, holger krekel <holger at merlinux.eu> wrote:
>> On Tue, Feb 15, 2011 at 10:12 +0900, Herman Sheremetyev wrote:
>>> I've seen a lot of messages on this list of people mentioning they use
>>> py.test, so I'm curious what advantages it offers. I gave it a try and
>>> immediately noticed it was literally 10x slower than nose, taking 0.60
>>> seconds to run the same set of tests it takes nose 0.062. As a result
>>> I pretty much wrote it off since test speed is pretty important to me,
>>> and all the red in my terminal seemed a bit unnecessary. But as I see
>>> people continuously mention it I wonder if I missed something about
>>> py.test's functionality that could be really useful and offset the
>>> speed problems.
>>
>> first a note on speed.  py.test has higher overhead for executing a single
>> tests than nose or unittest.  It spends more time in plugin hook invocations
>> and could be optimized to be less noticeable.  But this hasn't been a priority
>> because no one has yet told me it's of practical concern to them.
>> Is it in your case?
>
> Definitely. It seems like people that are using py.test are writing
> much slower tests, perhaps exercising large portions of their
> application in each one, so the extra setup time gets lost in the
> noise. Whereas if you have a set of tests each of which takes
> microseconds to run, running the test suite with unittest or nose
> doesn't break you out of the development groove, but py.test takes so
> long that I think it makes TDD style development much less practical.
> At least that's the case for me. If I had to worry about doing
> end-to-end testing in a large application, rather than doing quick
> iterations in a small library, py.test sounds like it might make for a
> good solution as it supports running the tests in parallel, which is
> key when you have lots of large tests.
>
>>> So, what are your reasons for using py.test (or some other test runner) ? :)
>>
>> i like many aspects of how it behaves and reports probably
>> because i wrote much of it :)  I'd also be curious what people
>> have to say why they prefer nose or other test runners, btw :)
>
> These are just visual nitpicks, but I find the default informational
> messages in the terminal being in red to be pretty confusing. Green,
> or perhaps just neutral bold, would have been a better choice for
> these I think. Also, the huge horizontal lines separating frames in
> the stack traces are very hard to parse visually. This may just be me
> though..
>
> Cheers,
>
> -Herman
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>