[TIP] how to test "impossible" events

[Andrew Dalke]

On Sep 30, 2008, at 1:12 AM, Grig Gheorghiu wrote:
> IMO this is overkill. The Web service that sends you data probably  
> relies on lower-level libraries/modules that put together the HTTP  
> responses. At this point, you're testing those low-level modules  
> and not your code.

I'm talking to a Tomcat server.  The back-end Java servlet was using  
the wrong port number to speak to another server, which somehow  
caused a NullPointerException.  This was converted into a SOAP error  
response, encoded as a MTOM message and send with an HTTP error  
status of 500 - "Internal Server Error".

Using the server you just pointed out:

 >>> import urllib2
 >>> f = urllib2.urlopen("http://httpstatus.appspot.com/500")
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
...
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ 
python2.5/urllib2.py", line 353, in _call_chain
     result = func(*args)
   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ 
python2.5/urllib2.py", line 499, in http_error_default
     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Internal Server Error
 >>>

I couldn't figure out how to suppress that error message - urllib/ 
urllib2 is a confusing mess of a module - so I ended up writing to  
the httplib module, which meant using some of the low-level modules.   
Since I had to write my own (limited) MTOM parser, it also ended up  
being easier with httplib than urllib2.


Because I don't have a full SOAP processor, I'm writing code to check  
the response content-type and other fields in case a server upgrade  
changes to another SOAP response type that I can't handle, or even  
just changes the order of the items in the response.

It turns out, btw, that Tomcat doesn't report the correct content- 
type, according to the MTOM spec.


> A more interesting test is what happens at the higher-level, i.e.  
> with the actual payload/data that your code receives. At that level  
> you can write unit tests against corner cases and see how your code  
> reacts to various abnormal data sets.

I can, but that's my question.  It seems to take a lot more work to  
set up automated tests for those not-known-to-exist corner cases than  
doing the manual tests that just check to see that the "raise  
AssertionError('something bad happened')"-type  code path doesn't  
have a typo.

> A better test in this case would be to verify that the Excel  
> document that you generate contains the data you expect.

Yeah, I haven't gotten to testing the Excel results yet, other than  
visual inspection.  How do I make sure that the text in column D  
which contains more than 5 characters is rotated vertically?

There are ways to do it, and perhaps this is simply bad habits of  
mine learned over long years of BASIC programming or some such  
reason, but it feels easier to check the output visually than trying  
to figure out some automated system system for this.  In any case,  
that's diverging from my question.

> Just verifying an exit code will not tell you anything terribly  
> exciting about the correctness of your code.


Checking the error code has proved useful before:

   - paths change and directories get reorganized. The exit code
       tells me if the program didn't start.

   - some programs use a time-based licenses (this one does) and if the
       license fails then it might report the failure through an exit  
code.
       As I found out today, this one doesn't work that way. :(
       At least I know the failure mode now - it sends text to stdout
       the does an exit(0).

    - I don't know what the program does on all cases of invalid
       input and the documentation isn't enough so I can't filter
       out all bad input from the program.  Some programs report
       input errors with exit(error_code).

Is that part of code correctness?  It feels like a pretty orthogonal  
topic to me.

> I personally find mock testing extremely beneficial for simulating  
> error conditions or exceptions that would otherwise be hard to  
> intercept reliably (since they tend to be random). So you could  
> have a mock Web service and simulate exceptions such as 'service  
> not available' or timeouts. Then you can test how your code reacts  
> to these exceptions.
>
> The other benefit of mock testing is to return known data (assuming  
> the live Web service returns data that has values that vary wildly  
> from one run to the other).

Perhaps I can rephrase my question.  I know how to write mock systems  
for all of the things I've talked about.  the question I've been  
thinking about as I write this code is the trade-off between:

   - do I write code to test for cases that aren't known to occur,
       based on experience that other similar code has done unexpected
       things during rare circumstances?

   - if so, do I test the code paths?  Should I

       - do manual tests that the code path executed suffice?

          (The tests are always similar to:
               if impossible_condition:
                 raise AssertionError("message: %s" % value)
          so there's very little that can go wrong with them.)

       - or should I automate those tests?

   - How much time should I spend setting up mock tests for
          something that isn't known to ever occur?

I know that, for example, SQLite has a filesystem isolation layer  
designed to test rare cases.  I'm quite impressed with the code  
coverage.  But at least some of those rare tests are still for errors  
that may occur and have occurred on some hardware.  I'm talking about  
for code where there's no evidence that a given failure mode will  
ever occur.

				Andrew
				dalke at dalkescientific.com