[TIP] how to test "impossible" events

[Andrew Dalke]

I've been developing some code for one of my clients.  I'm
going through the code, cleaning things up, removing dead
code, writing comments, etc.  My test suite is pretty minimal,
and I've been trying to think of how to make it better.

The code merges data from two web services and one command-line
program then uses Excel through COM to generate a spreadsheet
report.  Already you can see that there are few parts that
work in isolation.

Based on experience, external programs can fail in all sorts
of fun and exciting ways, and it's best to check for failures
that don't occur (or haven't yet occurred) in real life but
which I've seen fail in other cases.

For examples, I check that the HTTP headers have the right
content-type, that the response body contains the expected
structure, that the number of fields in the response matches
the number of fields I requested, etc.  For the command-line
program I check that it exits without an error code, and
I might add code to check that it sends nothing to stderr.

None of these paranoia checks fail against real-world data.
They are checking for some hypothetical future-world breakdown.

Does this count as YAGNI?

Worse yet, automated testing those paths has proved hard.  Manual
testing is easy, because I have code like this:


   if http_response_code != 200:
     raise TypeError("unexpected response code %r" %  
(http_response_code,))

and I can stick in

    http_response_code = 200

to make sure that the string gets interpolated correctly and
that the correct exception gets raised.  Once that happens,
I delete the code and don't touch the if-branch again.

Automated testing though - do I make a mock layer for urllib
and for subprocess.Popen?

When I've tried that approach I've found that it takes a lot
of time to write, especially as the actual Python code I'm
testing is only a few dozen lines, and because I'll have to
hand-write those failure cases.


				Andrew
				dalke at dalkescientific.com