[TIP] alphabetical order in testing
olemis at gmail.com
Wed Jan 20 10:43:55 PST 2010
On Wed, Jan 20, 2010 at 12:43 PM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
> Alfredo Deza wrote:
>> Can someone give me a good explanation *why* test methods are executed
>> alphabetically rather than the normal
>> Python way (top down). ?
> For all that many posters have claimed that order of tests should not
> matter, it does -- even for unit tests. You can _intend_ to implement your
> tests so that it does not matter which order they execute in, but what if
> you make a mistake and create a side-effect that influences the outcome of a
> later test?
Then you have *EXACTLY* a fragile test and that's an anti-pattern (for
many reasons ...). That's not supposed to happen and, in that case,
two actions should be taken:
- `tearDown` method should return things back to normal state. That's
what happens in Trac + `dutest` example. Even if the stub Trac
environment is not torn down completely it is emptied (DB included)
all doctests loaded from the same source and populated before
that happens. Result: it's quite unlikely to propagate side-effects .
- `setUp` should check that all the conditions needed to start the
test case have been met and, if not, report an *ERROR* (not a
*FAILURE* ;o) to indicate `tearDown` left things behind ...
It's something like design by contract but applied to the testing
process. In there you have preconditions, post-conditions, invariants
and recovery strategies . If tester wants to protect the test code
against those issues, that could be something reliable to consider
(not test order).
> In that case, it is useful to have a fixed order to make it
> easier to debug your tests.
Or run them at random, because perhaps there are interactions with
what's tested in one test case but not with the other . Eg
If you always run tests in the last order then you'll always have a
hidden bug, and quite probably weak test code ...
> And to get back to the original question:
>> Would it be OK to alter the naming in the methods to make sure they are
>> in the order I need to? Even if the naming gets really weird like:
Of course you can. Subclass TestSuite and override TestLoader.suiteClass
> As with most programming questions, the answer is "Yes, it is OK, but be
> sure that you understand the implications of what you are doing".
and be aware of the consequences ;o)
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
Added CHANGES.txt. Needed to execute `setup.py`. -
More information about the testing-in-python