[TIP] alphabetical order in testing

[holger krekel]

On Wed, Jan 20, 2010 at 06:52 -0800, C. Titus Brown wrote:
> On Wed, Jan 20, 2010 at 09:12:29AM -0500, Olemis Lang wrote:
> > On Wed, Jan 20, 2010 at 5:42 AM, holger krekel <holger at merlinux.eu> wrote:
> > > Some way of telling a test runner to consider tests in a certain
> > > order makes sense to me, though.
> > >
> > > In fact, I had user requests and own thoughts to allow declaring dependencies -
> > > such that if a test fails subsequent ones would not be run anymore.
> > 
> > Probably (I don't know the exact details ;o), but if the first test
> > fails that (should ?) means that something is not right (i.e. behaving
> > like expected) with the SUT .
> 
> Olemis, et al. -- independence of test order is generally considered a
> requirement for unit tests, but not for other tests (functional, UI, etc.)
> where it simply wouldn't make sense.
>
> If we're going to be dogmatic, let's be specific in our dogmatism, too :)
> 
> > Anyway : be pragmatic ! ... and be aware of the consequences ;o)
> 
> I think we can count on Holger to be aware of the consequences!

Consequences consequences - i am almost scared now :) 

Actually there are two features people are regularly using with py.test which
uncover wrong assumtions about test ordering:  "looponfailing" which
repeatedly runs only the failing test set in a sub process and none of
the passing ones.  And load-balancing where tests effectively run in 
somewhat random order in the sub processes.  

Actually, some time ago i broke file-position ordering - maybe it was
alphabetical then - and people came back complaining!  They *liked* it that
tests run in file order because it means the can have their test file editor
open and during refactoring fix their test one after another without much
jumping around.  As Grig puts it in his blog post [1]

     A feature of py.test which is a pleasant change from unittest is that the
     test execution order is guaranteed to be the same for each test run, and
     it is simply the order in which the test function/methods appear in a
     given test file.  No alphanumerical sorting order to worry about.

Maybe i should actually start getting more dogmatic, specifically so :) 

cheers,
holger

[1] http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-3-pytest-tool.html