[TIP] how to test "impossible" events

[Grig Gheorghiu]

Hi, Andrew!

--- On Mon, 9/29/08, Andrew Dalke <dalke at dalkescientific.com> wrote:

> 
> For examples, I check that the HTTP headers have the right
> content-type, that the response body contains the expected
> structure, that the number of fields in the response
> matches
> the number of fields I requested, etc.  

IMO this is overkill. The Web service that sends you data probably relies on lower-level libraries/modules that put together the HTTP responses. At this point, you're testing those low-level modules and not your code. A more interesting test is what happens at the higher-level, i.e. with the actual payload/data that your code receives. At that level you can write unit tests against corner cases and see how your code reacts to various abnormal data sets.


> For the
> command-line
> program I check that it exits without an error code, and
> I might add code to check that it sends nothing to stderr.

A better test in this case would be to verify that the Excel document that you generate contains the data you expect. Just verifying an exit code will not tell you anything terribly exciting about the correctness of your code.

> 
> None of these paranoia checks fail against real-world data.
> They are checking for some hypothetical future-world
> breakdown.
> 
> Does this count as YAGNI?

In a sense it is YAGNI. In another sense, see below for mock testing.

> 
> Automated testing though - do I make a mock layer for
> urllib
> and for subprocess.Popen?
> 
> When I've tried that approach I've found that it
> takes a lot
> of time to write, especially as the actual Python code
> I'm
> testing is only a few dozen lines, and because I'll
> have to
> hand-write those failure cases.
> 
> 

I personally find mock testing extremely beneficial for simulating error conditions or exceptions that would otherwise be hard to intercept reliably (since they tend to be random). So you could have a mock Web service and simulate exceptions such as 'service not available' or timeouts. Then you can test how your code reacts to these exceptions.

The other benefit of mock testing is to return known data (assuming the live Web service returns data that has values that vary wildly from one run to the other).

HTH,

Grig