[TIP] alphabetical order in testing

Olemis Lang olemis at gmail.com
Wed Jan 20 10:43:55 PST 2010


On Wed, Jan 20, 2010 at 12:43 PM, Mark Sienkiewicz <sienkiew at stsci.edu> wrote:
> Alfredo Deza wrote:
>>
>> Can someone give me a good explanation *why* test methods are executed
>> alphabetically  rather than the normal
>> Python way (top down). ?
>>
>>
>
[...]
>
> For all that many posters have claimed that order of tests should not
> matter, it does -- even for unit tests.  You can _intend_ to implement your
> tests so that it does not matter which order they execute in, but what if
> you make a mistake and create a side-effect that influences the outcome of a
> later test?

Then you have *EXACTLY* a fragile test and that's an anti-pattern (for
many reasons ...). That's not supposed to happen and, in that case,
two actions should be taken:

  - `tearDown` method should return things back to normal state. That's
    what happens in Trac + `dutest` example. Even if the stub Trac
    environment is not torn down completely it is emptied (DB included)
    after running
    all doctests loaded from the same source and populated before
    that happens. Result: it's quite unlikely to propagate side-effects .
  - `setUp` should check that all the conditions needed to start the
    test case have been met and, if not, report an *ERROR* (not a
    *FAILURE* ;o) to indicate `tearDown` left things behind ...

It's something like design by contract but applied to the testing
process. In there you have preconditions, post-conditions, invariants
and recovery strategies . If tester wants to protect the test code
against those issues, that could be something reliable to consider
(not test order).

> In that case, it is useful to have a fixed order to make it
> easier to debug your tests.
>

Or run them at random, because perhaps there are interactions with
what's tested in one test case but not with the other . Eg

{{{
test_a (ok)
test_b (ok)
test_c (failure)
}}}

{{{
test_a (ok)
test_c (ok)
test_b (ok)
}}}

If you always run tests in the last order then you'll always have a
hidden bug, and quite probably weak test code ...

>
> And to get back to the original question:
>
>> Would it be OK to alter the naming in the methods to make sure they are
>> run
>> in the order I need to? Even if the naming gets really weird like:
>

Of course you can. Subclass TestSuite and override TestLoader.suiteClass
;o)

> As with most programming questions, the answer is "Yes, it is OK, but be
> sure that you understand the implications of what you are doing".
>

and be aware of the consequences ;o)

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Added CHANGES.txt. Needed to execute `setup.py`.  -
http://flioops.hg.sourceforge.net/hgweb/flioops/dutest/rev/4e166ba04631



More information about the testing-in-python mailing list