[TIP] alphabetical order in testing
fuzzyman at voidspace.org.uk
Wed Jan 20 06:32:33 PST 2010
On 20/01/2010 14:27, Alfredo Deza wrote:
> On Wed, Jan 20, 2010 at 9:12 AM, Olemis Lang <olemis at gmail.com
> <mailto:olemis at gmail.com>> wrote:
> On Wed, Jan 20, 2010 at 5:42 AM, holger krekel <holger at merlinux.eu
> <mailto:holger at merlinux.eu>> wrote:
> > On Tue, Jan 19, 2010 at 15:35 -0500, Olemis Lang wrote:
> >> On Tue, Jan 19, 2010 at 3:26 PM, Alfredo Deza
> <alfredodeza at gmail.com <mailto:alfredodeza at gmail.com>> wrote:
> >> > Hi
> >> >
> >> > I am aware that test methods are executed in alphabetical order.
> >> >
> >> > What would be the ideal/preferable way of dealing with
> methods that depend
> >> > in the order they are being executed?
> >> >
> >> > Given that:
> >> >
> >> >> class TestFiles(unittest.TestCase):
> >> >>
> >> >> def test_second_file(self):
> >> >> """I depend on z_file"""
> >> >>
> >> >> def test_z_file(self):
> >> >> """I should run before second_file"""
> >> >
> >> >
> >> > Would it be OK to alter the naming in the methods to make
> sure they are run
> >> > in the order I need to? Even if the naming gets really weird
> >> >
> >> Very bad practice ! Dependencies between test cases can lead to
> >> fragile tests . TCs should be as isolated as possible
> >> ;o)
> > Some way of telling a test runner to consider tests in a certain
> > order makes sense to me, though.
> Can someone give me a good explanation *why* test methods are executed
> alphabetically rather than the normal
> Python way (top down). ?
How is that 'normal' in Python? Methods in a class are not ordered in
any way. In fact they are stored in a dictionary which is specifically
> > In fact, I had user requests and own thoughts to allow declaring
> dependencies -
> > such that if a test fails subsequent ones would not be run anymore.
> Probably (I don't know the exact details ;o), but if the first test
> fails that (should ?) means that something is not right (i.e. behaving
> like expected) with the SUT . If tester does not want to run the other
> tests thats because some assertions about the SUT have not been met
> rather than «test case X failed». I strongly suggest to write test
> cases by focusing on the SUT, and avoid dependencies between test
> (cases) . If the condition to skip the dependent test cases is exactly
> the same that will be asserted then I (a notoriously non-practical
> person that wants to sleep quietly one night after another) suggest to
> be explicit and create a test stub (function ? but not a test case)
> and write an assertion that just calls the test stub. OTOH if the
> framework has a skip support then use the same condition to skip the
> other test cases .
> Of course nothing like this will work with Py 2.7 skip functions since
> the skip decorators evaluate the condition at the wrong (IMO) time
> _ .
> There are also exceptions to the rule. For example, `dutest` _ maps
> a test case to each interactive example executed (instead of DocTest)
> . In that case someone (/me included) might say that when using
> REPORT_ONLY_FIRST_FAILURE there are some dependencies between TCs .
> But as you can see that's a very particular case and is conditioned by
> the characteristics of doctest. But XUnit-style is different and
> should not suffer with those symptoms.
> Anyway : be pragmatic ! ... and be aware of the consequences ;o)
> > py.test
> > runs tests in the order in which they appear in the file and i
> am considering
> > marking a test case/class as being "incremental", i.e. skip/fail
> tests after
> > the first tests fails.
> Even worse, why (if alphabetical is the correct way) test frameworks
> like py.test
> do them in a different way? This adds a whole layer of confusion to
> someone that
> wants to get started correctly. However I do agree that it is good to
> have options
> and different ways of doing things. But I also support conventions.
It doesn't matter what the default order of test runs are because it is
*well understood* that individual tests need to be independent of one
> Good luck !
> ..  Assessment of unittest 2.7 API : new features and opinions
> ..  dutest @ PyPI
> Blog ES: http://simelo-es.blogspot.com/
> Blog EN: http://simelo-en.blogspot.com/
> Featured article:
> Removing PyOOP test modules. -
> Alfredo Deza
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the testing-in-python