[TIP] alphabetical order in testing

[Olemis Lang]

On Wed, Jan 20, 2010 at 5:42 AM, holger krekel <holger at merlinux.eu> wrote:
> On Tue, Jan 19, 2010 at 15:35 -0500, Olemis Lang wrote:
>> On Tue, Jan 19, 2010 at 3:26 PM, Alfredo Deza <alfredodeza at gmail.com> wrote:
>> > Hi
>> >
>> > I am aware that test methods are executed in alphabetical order.
>> >
>> > What would be the ideal/preferable way of dealing with methods that depend
>> > in the order they are being executed?
>> >
>> > Given that:
>> >
>> >> class TestFiles(unittest.TestCase):
>> >>
>> >>     def test_second_file(self):
>> >>         """I depend on z_file"""
>> >>
>> >>     def test_z_file(self):
>> >>         """I should run before second_file"""
>> >
>> >
>> > Would it be OK to alter the naming in the methods to make sure they are run
>> > in the order I need to? Even if the naming gets really weird like:
>> >
>>
>> Very bad practice ! Dependencies between test cases can lead to
>> fragile tests . TCs should be as isolated as possible
>> ;o)
>
> Some way of telling a test runner to consider tests in a certain
> order makes sense to me, though.
>
> In fact, I had user requests and own thoughts to allow declaring dependencies -
> such that if a test fails subsequent ones would not be run anymore.

Probably (I don't know the exact details ;o), but if the first test
fails that (should ?) means that something is not right (i.e. behaving
like expected) with the SUT . If tester does not want to run the other
tests thats because some assertions about the SUT have not been met
rather than «test case X failed». I strongly suggest to write test
cases by focusing on the SUT, and avoid dependencies between test
(cases) . If the condition to skip the dependent test cases is exactly
the same that will be asserted then I (a notoriously non-practical
person that wants to sleep quietly one night after another) suggest to
be explicit and create a test stub (function ? but not a test case)
and write an assertion that just calls the test stub. OTOH if the
framework has a skip support then use the same condition to skip the
other test cases .

Of course nothing like this will work with Py 2.7 skip functions since
the skip decorators evaluate the condition at the wrong (IMO) time
[1]_ .

There are also exceptions to the rule. For example, `dutest` [2]_ maps
a test case to each interactive example executed (instead of DocTest)
. In that case someone (/me included) might say that when using
REPORT_ONLY_FIRST_FAILURE there are some dependencies between TCs .
But as you can see that's a very particular case and is conditioned by
the characteristics of doctest. But XUnit-style is different and
should not suffer with those symptoms.

Anyway : be pragmatic ! ... and be aware of the consequences ;o)

> py.test
> runs tests in the order in which they appear in the file and i am considering
> marking a test case/class as being "incremental", i.e. skip/fail tests after
> the first tests fails.
>

Good luck !

.. [1] Assessment of unittest 2.7 API : new features and opinions
        (http://simelo-en.blogspot.com/2009/12/assessment-of-unittest-27-api-new.html)

.. [2] dutest @ PyPI
        (http://pypi.python.org/pypi/dutest/0.2.3)

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Removing PyOOP test modules.  -
http://flioops.hg.sourceforge.net/hgweb/flioops/dutest/rev/acc55ed8a1c0