[TIP] py.test doesn't appear to consider testscenarios tests to be distinct tests

[Julian Berman]

Hi. I'd post this as a bug on one of the two relevant projects but I
haven't spent any time investigating and Robert nudged me to just post it
here :).

Consider the following test module:

### test.py
from unittest import TestCase
from testscenarios import WithScenarios


class TestTest(WithScenarios, TestCase):
    scenarios = [("one", {"foo" : 13}), ("two", {"foo" : 12})]

    def test_stuff(self):
        if self.foo == 12:
            self.fail()


Let's see what happens with each test runner :)

(relevant versions are 2.3.4 for pytest, 1.2.1 for nose, 12.3.0 for
Twisted, 2.7.3 for unittest)

⊙  py.test test.py

                                        Julian at air ●
===========================================================================================
test session starts
============================================================================================
platform darwin -- Python 2.7.3 -- pytest-2.3.4
collected 1 items

test.py F

=================================================================================================
FAILURES
=================================================================================================
___________________________________________________________________________________________
TestTest.test_stuff
____________________________________________________________________________________________

self = <test.TestTest testMethod=test_stuff>

    def test_stuff(self):
        if self.foo == 12:
>           self.fail()
E           AssertionError: None

test.py:10: AssertionError
=========================================================================================
1 failed in 0.28 seconds
=========================================================================================


py.test runs each scenario apparently, but considers them all to be one
test so failures are essentially impossible to debug. --collectonly shows
that py.test finds only "1 test".

trial does the right thing (yay :):
⊙  trial test

                                         Julian at air
test
  TestTest
    test_stuff(one) ...
 [OK]
    test_stuff(two) ...
 [FAIL]

===============================================================================
[FAIL]
Traceback (most recent call last):
  File
"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py",
line 327, in run
    testMethod()
  File "/Users/Julian/Desktop/test.py", line 10, in test_stuff
    self.fail()
  File
"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py",
line 408, in fail
    raise self.failureException(msg)
exceptions.AssertionError: None

test.TestTest.test_stuff(two)
-------------------------------------------------------------------------------
Ran 2 tests in 0.064s

FAILED (failures=1, successes=1)


and nose blows up (though I'm told this isn't really a surprise due to how
differently nose runs tests :):
⊙  nosetests

                                          Julian at air
E
======================================================================
ERROR: test_stuff (test.TestTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 133, in
run
    self.runTest(result)
  File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 151, in
runTest
    test(result)
AssertionError: ResultProxy for Test(<test.TestTest testMethod=test_stuff>)
(4374260944) was called with test <test.TestTest testMethod=test_stuff>
(4374299152)

----------------------------------------------------------------------
Ran 0 tests in 0.064s

FAILED (errors=1)

The unittest module also works, though apparently there's an outstanding
bug preventing it from working ideally:
⊙  python -m unittest -v test

                                         Julian at air
test_stuff (test.TestTest) ... ok
test_stuff (test.TestTest) ... FAIL

======================================================================
FAIL: test_stuff (test.TestTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 10, in test_stuff
    self.fail()
AssertionError: None

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)


Anyways, the surveying is idle fun, but do you (Holger :) or Robert happen
to have any idea what's going on with pytest here?

Cheers,
Julian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20130210/553c3868/attachment.htm>