[TIP] Supporting multiple fixtures for a test suite

[Titus Brown]

On Thu, Jan 03, 2008 at 03:26:29PM -0500, Gary Bernhardt wrote:
-> I have a fairly complex client-server system with a suite of
-> functional tests.  I want to be able to run the tests in two ways: by
-> spawning the server and running normally; or with paste.fixture, which
-> hooks the client directly to the WSGI app.  Supporting both test modes
-> allows me to make a tradeoff whenever I run the tests: do I want them
-> to be more complete (i.e., the full test mode with the server), or do
-> I want them to be fast (i.e., with the short-circuited WSGI mode,
-> which takes about half as long to run).  To put it simply, what I want
-> is a single test suite that can choose between multiple fixtures.
-> 
-> I've considered a couple solutions.  First, I could add a package that
-> imports all of the functional tests, but adds its own setup and
-> teardown methods.  I don't like this because the packages could
-> accidentally go out of sync (i.e., I forget to mirror a module or a
-> class from one to the other).  Second, I could add an environment
-> variable that chooses between the fixtures.  I don't like this either
-> because it's not obvious - you wouldn't know there are multiple
-> fixtures unless you dug down into the fixtures themselves.
-> 
-> So finally, the question I have is: how do you guys deal with
-> overloaded tests like this?  Is there some standard way to do it that
-> I've failed to google? :)

Not that I know of, although my guess is that the Zope folk have some
fantastic, but unproselytized, testing tools that probably do this
automatically by actually reading the mind of the original test coder
and doing what the person *actually* wanted to do.  But I digress...

Have you thought about importing the actual test code from a common
module, and providing different fixtures?  While I don't know that this
code actually works, it *should* work in nose ;)

---
# test-WSGI-only.py

from common_tests import test*

def setup():
  # start up paste.fixture

def teardown():
  # teardown paste.fixture

---
# test-live-site.py

from common_tests import test*

def setup():
   # start up live site

def teardown():
   # tear down live site
---

Once you have the tests installed with the different enclosing module
names, then it's simply a matter of choosing whether or not to run them.
How to do this depends on your testing framework, though.

I completely agree that NOT doing this sort of thing explicitly will
lead to you shooting yourself in your foot down the road...

cheers,
--titus