[TIP] Nose bug? using inspect.ismethod too aggressive?
Fernando Perez
fperez.net at gmail.com
Sun Sep 20 22:50:09 PDT 2009
Hi Rob,
On Sun, Sep 20, 2009 at 9:25 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> The functionality is nice, and its clearly superior to testscenarios for
> the sorts of parameterisation you're doing.
Thanks!
> However, the run_parametric function duplicates most of
> TestCase.__call__ which -will- lead to skew and bugs. In particular
> you're missing all of the 2.7/3.1 changes [some of which [class
> skipping] I hope to cleanup 'soon'] but most of which are really good
> and useful.
Yes, that is indeed very unfortunate. I tried other solutions that
would be less invasive, but they all failed for me, because I need to
trap the StopIteration exception immediately and record it as not
being an error but instead a normal exit.
I also tried to let it propagate and back-introspect the error after
the fact, but that was really hackish. Unfortunately in all my
encounters with unittest and doctest, 'extending' has required massive
copy/pasting, because they hold state all over the place in ways that
makes single-method customizations nearly impossible in all but the
most trivial problems.
> Looking at your implementation, you call setUp()/next()/tearDown(). This
> might cause some confusion if users cache attributes :). However,
> assuming thats deliberate, what about the below (it needs 2.7/3.1
> because of the improved TestSuite). Doing it for older pythons just
> needs a slightly larger TestSuite (you have to override a couple more
> methods to use self.__iter__).
Well, the problem is that I need to call startTest/stopTest once per
iteration, so each is (as it should) properly counted as a test. So I
decided to then let it also do setup/teardown as well, as it seemed
the consistent thing to do. But I also pondered instead doing
setup/teardown only once for the whole group, and then only running
the actual iteration. I suppose that if I do that and only call
startTest at the beginning of the iteration and not before setUp, it
could work.
Regarding your solution, it's not 100% obvious to me from just looking
at it that all that adaptation isn't going to get me back to where I
don't want to be: an incomprehensible stack for interactive debugging.
What I'm trying to get is to NOT interpose layers inside the test
method that change the execution context so much that debugging a
failing test becomes very hard.
With my approach, a failing test lets me do this:
maqroll[tests]> nosetests --pdb-failures test.py
> /home/fperez/code/python/ptest/tests/test.py(114)is_smaller()
-> assert i<j,"%s !< %s" % (i,j)
(Pdb) print i
2
(Pdb) print j
2
(Pdb) up
> /home/fperez/code/python/ptest/tests/test.py(127)test_par_standalone()
-> yield is_smaller(x, y)
(Pdb) print x
2
(Pdb) q
The stack I'm navigating is the stack of my test code. Today with a
nose test generator, the same thing lands you in the already-consumed
generator and the nose code:
maqroll[tests]> nosetests --pdb-failures test.py
> /home/fperez/code/python/ptest/tests/test.py(114)is_smaller()
-> assert i<j,"%s !< %s" % (i,j)
(Pdb) u
> /var/lib/python-support/python2.6/nose/case.py(182)runTest()
-> self.test(*self.arg)
(Pdb) l
177
178 def id(self):
179 return str(self)
180
181 def runTest(self):
182 -> self.test(*self.arg)
183
184 def shortDescription(self):
185 if hasattr(self.test, 'description'):
186 return self.test.description
187 func, arg = self._descriptors()
(Pdb) u
> /usr/lib/python2.6/unittest.py(279)run()
-> testMethod()
(Pdb) u
*** Oldest frame
You do have the first frame of the failed assertion, but everything
after that is already-evaluated static information, and the code that
produced self.arg is gone. So understanding the problem is nearly
impossible from interactive debugging (which I love, though I admit
I'm very biased towards such workflows).
In fact, I think your approach does have some of what I'm worried
about, if I understood you correctly and I didn't screw up. I fixed
up your code to actually run with 3.1 (note, I had to patch up
unittest.py quickly to prevent it from crashing with a None in the
test function which results below):
def isgenerator(func):
return hasattr(func,'_generator')
class IterCallableSuite(TestSuite):
def __init__(self, iterator, adapter):
self._iter = iterator
self._adapter = adapter
def __iter__(self):
for callable in self._iter:
yield self._adapter(callable)
class ParametricTestCase(unittest.TestCase):
"""Write parametric tests in normal unittest testcase form.
Limitations: the last iteration misses printing out a newline when
running in verbose mode.
"""
def run(self, result=None):
testMethod = getattr(self, self._testMethodName)
# For normal tests, we just call the base class and return that
if isgenerator(testMethod):
def adapter(next_test):
return unittest.FunctionTestCase(next_test,
self.setUp,
self.tearDown)
return IterCallableSuite(testMethod(),adapter).run(result)
else:
return super(ParametricTestCase, self).run(result)
def parametric(func):
"""Decorator to make a simple function into a normal test via
unittest."""
# Hack, until I figure out how to write isgenerator() for python3!!
func._generator = True
class Tester(ParametricTestCase):
test = staticmethod(func)
Tester.__name__ = func.__name__
return Tester
but when run it gives me:
maqroll[tests]> py3 test.py
....EE.
======================================================================
ERROR: unittest.FunctionTestCase (None)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.1/unittest.py", line 480, in run
testMethod()
File "/usr/local/lib/python3.1/unittest.py", line 1171, in runTest
self._testFunc()
TypeError: 'NoneType' object is not callable
======================================================================
ERROR: unittest.FunctionTestCase (None)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.1/unittest.py", line 480, in run
testMethod()
File "/usr/local/lib/python3.1/unittest.py", line 1171, in runTest
self._testFunc()
TypeError: 'NoneType' object is not callable
----------------------------------------------------------------------
Ran 7 tests in 0.003s
FAILED (errors=2)
One of those mystery tracebacks for the failed tests...
It would be great if we could make your idea work, because it's indeed
much cleaner code than what I posted. But I'm worried that there's
indeed a problem here:
def __iter__(self):
for callable in self._iter:
yield self._adapter(callable)
you are walking the iterator yourself, and the we can't do that.
Walking the iterator *is* running the test. Note the crucial
difference between how I spell a parametric test entry:
yield is_smaller(3, 4)
vs how it's done in nose:
yield (is_smaller,3, 4)
And I want #1, not #2, so debugging happens in *my* stack, not someone else's.
I'm pretty new to the intricacies of 3.1 and I don't know the new
unittest changes, so perhaps I just failed to adapt your code.
I went back once more and tried to change it this way:
def isgenerator(func):
return hasattr(func,'_generator')
class IterCallableSuite(TestSuite):
def __init__(self, iterator, adapter):
self._iter = iterator
self._adapter = adapter
def __iter__(self):
yield self._adapter(self._iter.__next__)
class ParametricTestCase(unittest.TestCase):
"""Write parametric tests in normal unittest testcase form.
Limitations: the last iteration misses printing out a newline when
running in verbose mode.
"""
def run(self, result=None):
testMethod = getattr(self, self._testMethodName)
# For normal tests, we just call the base class and return that
if isgenerator(testMethod):
def adapter(next_test):
return unittest.FunctionTestCase(next_test,
self.setUp,
self.tearDown)
return IterCallableSuite(testMethod(),adapter).run(result)
else:
return super(ParametricTestCase, self).run(result)
def parametric(func):
"""Decorator to make a simple function into a normal test via
unittest."""
# Hack, until I figure out how to write isgenerator() for python3!!
func._generator = True
class Tester(ParametricTestCase):
test = staticmethod(func)
Tester.__name__ = func.__name__
return Tester
Now it does actually run to completion, but it fails to correctly
iterate the tests:
## PYTHON 3.1:
maqroll[tests]> py3 test.py -v
test_parametric (__main__.Tester) ... ok
test (ptest.decotest.ipdt_flush) ... ok
test (ptest.decotest.ipdt_indented_test) ... ok
test (ptest.decotest.simple_dt) ... ok
x.__next__() <==> next(x) ... ok ### WHAT IS THIS ???
test (ptest.decotest.trivial) ... ok
----------------------------------------------------------------------
Ran 6 tests in 0.002s
OK
## PYTHON 2.6:
maqroll[tests]> python test.py -v
test_parametric (__main__.Tester) ... ok
test_parametric (__main__.Tester) ... ok
test_parametric (__main__.Tester) ... test (ptest.decotest.ipdt_flush) ... ok
test (ptest.decotest.ipdt_indented_test) ... ok
test (ptest.decotest.simple_dt) ... ok
test (ptest.decotest.test_par_standalone) ... ok
test (ptest.decotest.test_par_standalone) ... ok
test (ptest.decotest.test_par_standalone) ... test
(ptest.decotest.trivial) ... ok
----------------------------------------------------------------------
Ran 10 tests in 0.007s
OK
Your setup really looks a lot better than what I did, so I hope that
someone who is more familiar with py3 than me can make it work. I
wouldn't mind carrying around some extra backported code for a while
to use that if it's going to be the long-term solution.
Ultimately I'd love to have this as an out-of-the-box supported
feature in unittest itself, so I hope we can figure it out :)
Many thanks for your help and interest!
Cheers,
f
More information about the testing-in-python
mailing list