[TIP] stop suite after failure

[loigu]

Kumar McMillan wrote:
> On Jan 3, 2008 5:54 AM, loigu <loigu at volny.cz> wrote:
>   
>> Thanks for your response,
>>
>> The approach with startTest doesn't work -- returning True just prevents
>> other plugins from seeing the test start, doesn't prevent test itself
>> from starting.
>>     
>
> Hmm, I think there must be a way to have a plugin stop the test.  Did
> you try the want* methods?  Like:
>
> class FailureThreshold(Plugin):
>     # ...as defined above
>     def _wantTestObject(self, test):
>         level = getattr(test, 'threshold', None)
>         if level is not None and level >= self.failure_threshold:
>             return False
>     wantClass = _wantTestObject
>     wantFunction = _wantTestObject
>     wantMethod = _wantTestObject
>   

Yes you can do this but BEFORE any test run. So if you want to run test
depending on others results using want_ methods it forces you to run one
test after another manually.

I know that it's rather rare to want this feature in unit testing (unit
tests should be small and fast - so why bother) but when small unit test
runs above distributed filesystem... :/

> ...if that doesn't work I'd suggest posting on the nose list and after
> that submitting a feature request ;)  I think a plugin should be able
> to stop a test based on a test's attributes.
>
> Kumar
>
>   
>> Currently I use this approach:
>> I had subclassed nose.suite.ContextSuite and replaced the run method
>> with something like
>>
>> ....
>> ....
>> #main loop
>>                 try:
>>                     for test in self._tests:
>>                         if result.shouldStop:
>>                             log.debug("stopping")
>>                             break
>>                         # each nose.case.Test will create its own result
>> proxy
>>                         # so the cases need the original result, to
>> avoid proxy
>>                         # chains
>>                         test(orig)
>> #here is the only difference against original function
>>
>>                         if getattr(test,'stopContext', None):
>>                             break;
>>                 finally:
>>                     self.has_run = True
>> ....
>> ....
>>
>> Then in plugin I'm generating the context suite with tests ordered and
>> in handleFailure and handleError I'm setting the stopContext attribute.
>>
>>
>> This approach works but it's not as maintainable as i would like to.
>>
>> All the best,
>>
>> Jiri Zouhar
>>
>>
>>
>> Kumar McMillan wrote:
>>     
>>> On Dec 29, 2007 4:45 PM, loigu <loigu at volny.cz> wrote:
>>>
>>>       
>>>> Hi,
>>>>
>>>> Does anybody know if it is possible to stop context suite execution
>>>> after test failure without preventing other suites to run?
>>>>
>>>> What I'm trying to do:
>>>>
>>>> I have test class containing some tests (trivial, but with non-trivial
>>>> time consumption).
>>>> There can be defined ordering on them (if test N fails, all tests > N
>>>> fails too).
>>>> So in this case it would be nice to stop after failure and do not run
>>>> other tests from the same class (we know that they will fail),
>>>> but continue in execution of other test suites.
>>>>
>>>> Is it there some easy or preferred way to do this?
>>>> Or it is nonsense to do such a thing?
>>>>
>>>>         
>>> One way you could do this is to use nose
>>> (http://somethingaboutorange.com/mrl/projects/nose/).  You would
>>> assign attributes to your tests to define the level and create a
>>> custom plugin to skip tests when the threshold has been reached.
>>>
>>> Here is some more info on writing nose plugins:
>>> http://somethingaboutorange.com/mrl/projects/nose/doc/writing_plugins.html
>>> http://somethingaboutorange.com/mrl/projects/nose/doc/plugin_interface.html
>>>
>>> You'd assign attributes like so:
>>>
>>> def test_beans():
>>>     assert 1==2
>>> test_beans.threshold = 5
>>>
>>> def test_rice():
>>>     # relates somehow to beans
>>>     assert 1==3
>>> test_rice.threshold = 6
>>>
>>> Then a plugin would look something like this:
>>>
>>> from nose.plugins import Plugin
>>> class FailureThreshold(Plugin):
>>>     name = "failure-threshold"
>>>     def configure(self, options, conf):
>>>         Plugin.configure(self, options, conf)
>>>         if not self.enabled:
>>>             return
>>>         self.failure_threshold = 0
>>>
>>>     def addError(self, test, err):
>>>         level = getattr(test, 'threshold', None)
>>>         if level is not None and level < self.failure_threshold:
>>>             # honor the lowest failing level
>>>             self.failure_threshold = level
>>>     addFailure = addError
>>>
>>>     def startTest(self, test):
>>>         level = getattr(test, 'threshold', None)
>>>         if level is not None and level >= self.failure_threshold:
>>>             return True # prevents the test from running, *I think*
>>>
>>>
>>>
>>> ...untested, of course.  However, if you are creating tests that
>>> depend on another test in some way I would suggest trying to decouple
>>> that dependency.  You can probably do so by inheriting common
>>> setUp/tearDown methods.  Otherwise, if you are just trying to make a
>>> logical guess as to what might start failing next then your approach
>>> sounds like it might save some cpu time.  You may also find that
>>> manually declaring "levels" will become fragile and error prone so if
>>> you can find a way to introspect the level instead, that would
>>> probably be better.
>>>
>>>
>>>
>>>
>>>
>>>
>>>       
>>>> Thanks,
>>>>
>>>> Jiri Zouhar
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> testing-in-python mailing list
>>>> testing-in-python at lists.idyll.org
>>>> http://lists.idyll.org/listinfo/testing-in-python
>>>>
>>>>
>>>>         
>>> _______________________________________________
>>> testing-in-python mailing list
>>> testing-in-python at lists.idyll.org
>>> http://lists.idyll.org/listinfo/testing-in-python
>>>
>>>
>>>
>>>       
>> _______________________________________________
>> testing-in-python mailing list
>> testing-in-python at lists.idyll.org
>> http://lists.idyll.org/listinfo/testing-in-python
>>
>>     
>
>
>