[TIP] Nose bug? using inspect.ismethod too aggressive?

[Fernando Perez]

2009/9/20 Robert Collins <robertc at robertcollins.net>:
> On Sat, 2009-09-19 at 23:05 -0700, Fernando Perez wrote:
>> Hi all,
>> ...
>
>> It seems to me that this is a nose bug, fixable with:
>>
>> -            if not ismethod(item):
>> +            if not hasattr(item,'__call__'):
>>
>> I'll try to monkeypatch it in the meantime, but it would be great to
>> have this done upstream, if it's indeed seen as a bug.
>
> if not callable(item):
>   ...
>
> would be better, I think?

callable() went away in Py3, so we might as well get used to the new
idioms.  No point now in writing deliberately 2.x-only code.  The
above was Guido's suggested new 'spelling' for callable(), if my
memory of his py3k talk serves me right, though I'd be happy to be
corrected.

> Also, is your code around? testscenarios is my alternative-approach to
> test parameterisation, I'd like to see how you made generators
> compatible with unittest. I'm guessing a TestSuite that acts as an
> adapter.

A simple modified run() method in a class I inherit from:


class ParametricTestCase(unittest.TestCase):
    """Write parametric tests in normal unittest testcase form.

    Limitations: the last iteration misses printing out a newline when running
    in verbose mode.
    """
    def run_parametric(self, result, testMethod):
        # But if we have a test generator, we iterate it ourselves
        testgen = testMethod()
        while True:
            try:
                # Initialize test
                result.startTest(self)

                # SetUp
                try:
                    self.setUp()
                except KeyboardInterrupt:
                    raise
                except:
                    result.addError(self, self._exc_info())
                    return
                # Test execution
                ok = False
                try:
                    testgen.next()
                    ok = True
                except StopIteration:
                    # We stop the loop
                    break
                except self.failureException:
                    result.addFailure(self, self._exc_info())
                except KeyboardInterrupt:
                    raise
                except:
                    result.addError(self, self._exc_info())
                # TearDown
                try:
                    self.tearDown()
                except KeyboardInterrupt:
                    raise
                except:
                    result.addError(self, self._exc_info())
                    ok = False
                if ok: result.addSuccess(self)

            finally:
                result.stopTest(self)

    def run(self, result=None):
        if result is None:
            result = self.defaultTestResult()
        testMethod = getattr(self, self._testMethodName)
        # For normal tests, we just call the base class and return that
        if isgenerator(testMethod):
            return self.run_parametric(result, testMethod)
        else:
            return super(ParametricTestCase, self).run(result)


def parametric(func):
    """Decorator to make a simple function into a normal test via unittest."""
    class Tester(ParametricTestCase):
        test = staticmethod(func)

    Tester.__name__ = func.func_name

    return Tester

Along with the decorator, this lets me write:


def is_smaller(i,j):
    assert i<j,"%s !< %s" % (i,j)

@parametric
def test_par_standalone():
    yield is_smaller(3, 4)
    x, y = 2, 2  #BREAKS HERE!
    yield is_smaller(x, y)

and this will be trivial to debug.

Granted, I have to use a decorator explicitly, but the benefit for me
is absolutely enormous: 100% compatibility with plain unittest, and
more importantly, I can debug these interactively with
--pdb/--pdb-failures, which is near nigh impossible with all the other
approaches I've seen for parametric tests, including nose's.  Since
they consume the generator early, by the time the test fails, all your
original execution context is gone, and debugging is pure hell.  For
me, if tests aren't *super easy* to immediately debug interactively on
failures/errors, they become nearly useless.

I've fought with this for a long time and only this weekend I realized
the solution was actually very simple, and could be done completely
within the unittest machinery with the tiny modification to run()
above.  It may have been done already elsewhere, but so far I'd seen
an old solution of mine that went into numpy, one from twisted.trial,
the one in nose and the one in py.test, and they all left me wanting.
This one, I'm finally happy with, I hope someone else finds it useful.
 At the very least, it makes *me* happy :)

I need to polish this up and it will go into IPython very soon, I'd
love for it (or something like it) to be ultimately available in
unittest itself, if others find it useful.

Cheers,

f