[TIP] stop suite after failure

[Kumar McMillan]

On Jan 3, 2008 5:54 AM, loigu <loigu at volny.cz> wrote:
> Thanks for your response,
>
> The approach with startTest doesn't work -- returning True just prevents
> other plugins from seeing the test start, doesn't prevent test itself
> from starting.

Hmm, I think there must be a way to have a plugin stop the test.  Did
you try the want* methods?  Like:

class FailureThreshold(Plugin):
    # ...as defined above
    def _wantTestObject(self, test):
        level = getattr(test, 'threshold', None)
        if level is not None and level >= self.failure_threshold:
            return False
    wantClass = _wantTestObject
    wantFunction = _wantTestObject
    wantMethod = _wantTestObject

...if that doesn't work I'd suggest posting on the nose list and after
that submitting a feature request ;)  I think a plugin should be able
to stop a test based on a test's attributes.

Kumar

>
> Currently I use this approach:
> I had subclassed nose.suite.ContextSuite and replaced the run method
> with something like
>
> ....
> ....
> #main loop
>                 try:
>                     for test in self._tests:
>                         if result.shouldStop:
>                             log.debug("stopping")
>                             break
>                         # each nose.case.Test will create its own result
> proxy
>                         # so the cases need the original result, to
> avoid proxy
>                         # chains
>                         test(orig)
> #here is the only difference against original function
>
>                         if getattr(test,'stopContext', None):
>                             break;
>                 finally:
>                     self.has_run = True
> ....
> ....
>
> Then in plugin I'm generating the context suite with tests ordered and
> in handleFailure and handleError I'm setting the stopContext attribute.
>
>
> This approach works but it's not as maintainable as i would like to.
>
> All the best,
>
> Jiri Zouhar
>
>
>
> Kumar McMillan wrote:
> > On Dec 29, 2007 4:45 PM, loigu <loigu at volny.cz> wrote:
> >
> >> Hi,
> >>
> >> Does anybody know if it is possible to stop context suite execution
> >> after test failure without preventing other suites to run?
> >>
> >> What I'm trying to do:
> >>
> >> I have test class containing some tests (trivial, but with non-trivial
> >> time consumption).
> >> There can be defined ordering on them (if test N fails, all tests > N
> >> fails too).
> >> So in this case it would be nice to stop after failure and do not run
> >> other tests from the same class (we know that they will fail),
> >> but continue in execution of other test suites.
> >>
> >> Is it there some easy or preferred way to do this?
> >> Or it is nonsense to do such a thing?
> >>
> >
> > One way you could do this is to use nose
> > (http://somethingaboutorange.com/mrl/projects/nose/).  You would
> > assign attributes to your tests to define the level and create a
> > custom plugin to skip tests when the threshold has been reached.
> >
> > Here is some more info on writing nose plugins:
> > http://somethingaboutorange.com/mrl/projects/nose/doc/writing_plugins.html
> > http://somethingaboutorange.com/mrl/projects/nose/doc/plugin_interface.html
> >
> > You'd assign attributes like so:
> >
> > def test_beans():
> >     assert 1==2
> > test_beans.threshold = 5
> >
> > def test_rice():
> >     # relates somehow to beans
> >     assert 1==3
> > test_rice.threshold = 6
> >
> > Then a plugin would look something like this:
> >
> > from nose.plugins import Plugin
> > class FailureThreshold(Plugin):
> >     name = "failure-threshold"
> >     def configure(self, options, conf):
> >         Plugin.configure(self, options, conf)
> >         if not self.enabled:
> >             return
> >         self.failure_threshold = 0
> >
> >     def addError(self, test, err):
> >         level = getattr(test, 'threshold', None)
> >         if level is not None and level < self.failure_threshold:
> >             # honor the lowest failing level
> >             self.failure_threshold = level
> >     addFailure = addError
> >
> >     def startTest(self, test):
> >         level = getattr(test, 'threshold', None)
> >         if level is not None and level >= self.failure_threshold:
> >             return True # prevents the test from running, *I think*
> >
> >
> >
> > ...untested, of course.  However, if you are creating tests that
> > depend on another test in some way I would suggest trying to decouple
> > that dependency.  You can probably do so by inheriting common
> > setUp/tearDown methods.  Otherwise, if you are just trying to make a
> > logical guess as to what might start failing next then your approach
> > sounds like it might save some cpu time.  You may also find that
> > manually declaring "levels" will become fragile and error prone so if
> > you can find a way to introspect the level instead, that would
> > probably be better.
> >
> >
> >
> >
> >
> >
> >> Thanks,
> >>
> >> Jiri Zouhar
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> testing-in-python mailing list
> >> testing-in-python at lists.idyll.org
> >> http://lists.idyll.org/listinfo/testing-in-python
> >>
> >>
> >
> > _______________________________________________
> > testing-in-python mailing list
> > testing-in-python at lists.idyll.org
> > http://lists.idyll.org/listinfo/testing-in-python
> >
> >
> >
>
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>