[TIP] alphabetical order in testing
Olemis Lang
olemis at gmail.com
Wed Jan 20 10:43:55 PST 2010
On Wed, Jan 20, 2010 at 12:43 PM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
> Alfredo Deza wrote:
>>
>> Can someone give me a good explanation *why* test methods are executed
>> alphabetically rather than the normal
>> Python way (top down). ?
>>
>>
>
[...]
>
> For all that many posters have claimed that order of tests should not
> matter, it does -- even for unit tests. You can _intend_ to implement your
> tests so that it does not matter which order they execute in, but what if
> you make a mistake and create a side-effect that influences the outcome of a
> later test?
Then you have *EXACTLY* a fragile test and that's an anti-pattern (for
many reasons ...). That's not supposed to happen and, in that case,
two actions should be taken:
- `tearDown` method should return things back to normal state. That's
what happens in Trac + `dutest` example. Even if the stub Trac
environment is not torn down completely it is emptied (DB included)
after running
all doctests loaded from the same source and populated before
that happens. Result: it's quite unlikely to propagate side-effects .
- `setUp` should check that all the conditions needed to start the
test case have been met and, if not, report an *ERROR* (not a
*FAILURE* ;o) to indicate `tearDown` left things behind ...
It's something like design by contract but applied to the testing
process. In there you have preconditions, post-conditions, invariants
and recovery strategies . If tester wants to protect the test code
against those issues, that could be something reliable to consider
(not test order).
> In that case, it is useful to have a fixed order to make it
> easier to debug your tests.
>
Or run them at random, because perhaps there are interactions with
what's tested in one test case but not with the other . Eg
{{{
test_a (ok)
test_b (ok)
test_c (failure)
}}}
{{{
test_a (ok)
test_c (ok)
test_b (ok)
}}}
If you always run tests in the last order then you'll always have a
hidden bug, and quite probably weak test code ...
>
> And to get back to the original question:
>
>> Would it be OK to alter the naming in the methods to make sure they are
>> run
>> in the order I need to? Even if the naming gets really weird like:
>
Of course you can. Subclass TestSuite and override TestLoader.suiteClass
;o)
> As with most programming questions, the answer is "Yes, it is OK, but be
> sure that you understand the implications of what you are doing".
>
and be aware of the consequences ;o)
--
Regards,
Olemis.
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Featured article:
Added CHANGES.txt. Needed to execute `setup.py`. -
http://flioops.hg.sourceforge.net/hgweb/flioops/dutest/rev/4e166ba04631
More information about the testing-in-python
mailing list