<div dir="ltr">Hi. I'd post this as a bug on one of the two relevant projects but I haven't spent any time investigating and Robert nudged me to just post it here :).<div><br></div><div style>Consider the following test module:</div>
<div style><br></div><div style>### test.py</div><div style><div>from unittest import TestCase</div><div>from testscenarios import WithScenarios</div><div><br></div><div><br></div><div>class TestTest(WithScenarios, TestCase):</div>
<div> scenarios = [("one", {"foo" : 13}), ("two", {"foo" : 12})]</div><div><br></div><div> def test_stuff(self):</div><div> if self.foo == 12:</div><div> self.fail()</div>
<div><br></div><div><br></div><div style>Let's see what happens with each test runner :)</div><div style><br></div><div style>(relevant versions are 2.3.4 for pytest, 1.2.1 for nose, 12.3.0 for Twisted, 2.7.3 for unittest)</div>
<div style><br></div><div style><div>⊙ py.test test.py Julian@air ●</div>
<div>=========================================================================================== test session starts ============================================================================================</div><div>platform darwin -- Python 2.7.3 -- pytest-2.3.4</div>
<div>collected 1 items </div><div><br></div><div>test.py F</div><div><br></div><div>================================================================================================= FAILURES =================================================================================================</div>
<div>___________________________________________________________________________________________ TestTest.test_stuff ____________________________________________________________________________________________</div><div><br>
</div><div>self = <test.TestTest testMethod=test_stuff></div><div><br></div><div> def test_stuff(self):</div><div> if self.foo == 12:</div><div>> self.fail()</div><div>E AssertionError: None</div>
<div><br></div><div>test.py:10: AssertionError</div><div>========================================================================================= 1 failed in 0.28 seconds =========================================================================================</div>
<div><br></div><div><br></div><div style>py.test runs each scenario apparently, but considers them all to be one test so failures are essentially impossible to debug. --collectonly shows that py.test finds only "1 test".</div>
<div style><br></div><div style>trial does the right thing (yay :):</div><div style><div>⊙ trial test Julian@air</div>
<div>test</div><div> TestTest</div><div> test_stuff(one) ... [OK]</div><div> test_stuff(two) ... [FAIL]</div><div>
<br></div><div>===============================================================================</div><div>[FAIL]</div><div>Traceback (most recent call last):</div><div> File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 327, in run</div>
<div> testMethod()</div><div> File "/Users/Julian/Desktop/test.py", line 10, in test_stuff</div><div> self.fail()</div><div> File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 408, in fail</div>
<div> raise self.failureException(msg)</div><div>exceptions.AssertionError: None</div><div><br></div><div>test.TestTest.test_stuff(two)</div><div>-------------------------------------------------------------------------------</div>
<div>Ran 2 tests in 0.064s</div><div><br></div><div>FAILED (failures=1, successes=1)</div><div><br></div><div><br></div><div style>and nose blows up (though I'm told this isn't really a surprise due to how differently nose runs tests :):</div>
<div style><div>⊙ nosetests Julian@air</div>
<div>E</div><div>======================================================================</div><div>ERROR: test_stuff (test.TestTest)</div><div>----------------------------------------------------------------------</div><div>
Traceback (most recent call last):</div><div> File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 133, in run</div><div> self.runTest(result)</div><div> File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest</div>
<div> test(result)</div><div>AssertionError: ResultProxy for Test(<test.TestTest testMethod=test_stuff>) (4374260944) was called with test <test.TestTest testMethod=test_stuff> (4374299152)</div><div><br></div>
<div>----------------------------------------------------------------------</div><div>Ran 0 tests in 0.064s</div><div><br></div><div>FAILED (errors=1)</div><div><br></div><div style>The unittest module also works, though apparently there's an outstanding bug preventing it from working ideally:</div>
<div style><div>⊙ python -m unittest -v test Julian@air</div>
<div>test_stuff (test.TestTest) ... ok</div><div>test_stuff (test.TestTest) ... FAIL</div><div><br></div><div>======================================================================</div><div>FAIL: test_stuff (test.TestTest)</div>
<div>----------------------------------------------------------------------</div><div>Traceback (most recent call last):</div><div> File "test.py", line 10, in test_stuff</div><div> self.fail()</div><div>AssertionError: None</div>
<div><br></div><div>----------------------------------------------------------------------</div><div>Ran 2 tests in 0.001s</div><div><br></div><div>FAILED (failures=1)</div><div><br></div><div><br></div><div style>Anyways, the surveying is idle fun, but do you (Holger :) or Robert happen to have any idea what's going on with pytest here?</div>
<div style><br></div><div style>Cheers,</div><div style>Julian</div></div></div></div></div></div></div>