[TIP] The test suite that was too dangerous to live

[Ben Finney]

Howdy all,

Summary: I need some help. I hope in the process of rescuing my test
suite, my learning can help others too.


Testing has, without a doubt, blossomed in my development workflow. One
of the main benefits I get from behaviour-driven development is
*confidence*: I can see every time I make a change which tests pass, so
I have a much better idea of how many anticipated mistakes my code is
currently avoiding.

I'm working on a project (‘python-daemon’) whose tests involve a lot of
situational test case setup: inter-process relations, race conditions,
polymorphic classes, non-deterministic behaviours. With the help of mock
objects (made *very* simple by the beautiful ‘MiniMock’ library), I'm
able to tame these environmental hazards, set up a fake environment, and
assert the behaviour my code should have on its environment.

The code base isn't really large or difficult; the unit test suite
should be comprehensible. This past week, though, I've realised that I
have created a monster.

Needing to make a couple of changes to the API and underlying behaviour,
I started in on the test suite, making changes in the expectations. Over
the past several months they've grown into rather a forest, with many
layers of indirection, dwarfing the size of the code they're testing;
everything I did to my tests had unexpected consequences elsewhere in
the test suite, leaving me scratching my head at the lack of insulation
between test cases.

Today I had a chilling realisation:

I no longer trust my tests.

I have wrestled the code base such that the tests now assert the new
behaviour and all pass, so I've released the code. But the strange test
suite behaviour I saw when making the modifications means I no longer
have the confidence that my tests are *actually* testing what I *think*
they're testing.

Perhaps what my code needs is a few fresh sets of eyes; it might be
clear to someone else where the best place to start improving the test
suite would be. Please get the sdist for ‘python-daemon’ 1.5.1 from PyPI
<URL:http://pypi.python.org/pypi/python-daemon/1.5.1/> and have a look
at the unit test suite.

I'd be quite happy to have this discussed openly here, as a public
exercise in how to rescue a unit test suite that's starting to become
too unwieldy, as a way of making it a benefit to more than just me.

-- 
 \         “Pinky, are you pondering what I'm pondering?” “I think so, |
  `\   Brain, but if it was only supposed to be a three hour tour, why |
_o__)   did the Howells bring all their money?” —_Pinky and The Brain_ |
Ben Finney