[TIP] alphabetical order in testing

Alfredo Deza alfredodeza at gmail.com
Wed Jan 20 06:27:58 PST 2010


On Wed, Jan 20, 2010 at 9:12 AM, Olemis Lang <olemis at gmail.com> wrote:

> On Wed, Jan 20, 2010 at 5:42 AM, holger krekel <holger at merlinux.eu> wrote:
> > On Tue, Jan 19, 2010 at 15:35 -0500, Olemis Lang wrote:
> >> On Tue, Jan 19, 2010 at 3:26 PM, Alfredo Deza <alfredodeza at gmail.com>
> wrote:
> >> > Hi
> >> >
> >> > I am aware that test methods are executed in alphabetical order.
> >> >
> >> > What would be the ideal/preferable way of dealing with methods that
> depend
> >> > in the order they are being executed?
> >> >
> >> > Given that:
> >> >
> >> >> class TestFiles(unittest.TestCase):
> >> >>
> >> >>     def test_second_file(self):
> >> >>         """I depend on z_file"""
> >> >>
> >> >>     def test_z_file(self):
> >> >>         """I should run before second_file"""
> >> >
> >> >
> >> > Would it be OK to alter the naming in the methods to make sure they
> are run
> >> > in the order I need to? Even if the naming gets really weird like:
> >> >
> >>
> >> Very bad practice ! Dependencies between test cases can lead to
> >> fragile tests . TCs should be as isolated as possible
> >> ;o)
> >
> > Some way of telling a test runner to consider tests in a certain
> > order makes sense to me, though.
>

Can someone give me a good explanation *why* test methods are executed
alphabetically  rather than the normal
Python way (top down). ?




> >
> > In fact, I had user requests and own thoughts to allow declaring
> dependencies -
> > such that if a test fails subsequent ones would not be run anymore.
>
> Probably (I don't know the exact details ;o), but if the first test
> fails that (should ?) means that something is not right (i.e. behaving
> like expected) with the SUT . If tester does not want to run the other
> tests thats because some assertions about the SUT have not been met
> rather than «test case X failed». I strongly suggest to write test
> cases by focusing on the SUT, and avoid dependencies between test
> (cases) . If the condition to skip the dependent test cases is exactly
> the same that will be asserted then I (a notoriously non-practical
> person that wants to sleep quietly one night after another) suggest to
> be explicit and create a test stub (function ? but not a test case)
> and write an assertion that just calls the test stub. OTOH if the
> framework has a skip support then use the same condition to skip the
> other test cases .
>
> Of course nothing like this will work with Py 2.7 skip functions since
> the skip decorators evaluate the condition at the wrong (IMO) time
> [1]_ .
>
> There are also exceptions to the rule. For example, `dutest` [2]_ maps
> a test case to each interactive example executed (instead of DocTest)
> . In that case someone (/me included) might say that when using
> REPORT_ONLY_FIRST_FAILURE there are some dependencies between TCs .
> But as you can see that's a very particular case and is conditioned by
> the characteristics of doctest. But XUnit-style is different and
> should not suffer with those symptoms.
>
> Anyway : be pragmatic ! ... and be aware of the consequences ;o)
>
> > py.test
> > runs tests in the order in which they appear in the file and i am
> considering
> > marking a test case/class as being "incremental", i.e. skip/fail tests
> after
> > the first tests fails.
> >
>

Even worse, why (if alphabetical is the correct way) test frameworks like
py.test
do them in a different way? This adds a whole layer of confusion to someone
that
wants to get started correctly. However I do agree that it is good to have
options
and different ways of doing things. But I also support conventions.


>
> Good luck !
>
> .. [1] Assessment of unittest 2.7 API : new features and opinions
>        (
> http://simelo-en.blogspot.com/2009/12/assessment-of-unittest-27-api-new.html
> )
>
> .. [2] dutest @ PyPI
>        (http://pypi.python.org/pypi/dutest/0.2.3)
>
> --
> Regards,
>
> Olemis.
>
> Blog ES: http://simelo-es.blogspot.com/
> Blog EN: http://simelo-en.blogspot.com/
>
> Featured article:
> Removing PyOOP test modules.  -
> http://flioops.hg.sourceforge.net/hgweb/flioops/dutest/rev/acc55ed8a1c0
>



-- 
Alfredo Deza
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20100120/f07fd444/attachment.html>


More information about the testing-in-python mailing list