[TIP] how to test "impossible" events
dalke at dalkescientific.com
Mon Sep 29 18:29:56 PDT 2008
On Sep 30, 2008, at 1:12 AM, Grig Gheorghiu wrote:
> IMO this is overkill. The Web service that sends you data probably
> relies on lower-level libraries/modules that put together the HTTP
> responses. At this point, you're testing those low-level modules
> and not your code.
I'm talking to a Tomcat server. The back-end Java servlet was using
the wrong port number to speak to another server, which somehow
caused a NullPointerException. This was converted into a SOAP error
response, encoded as a MTOM message and send with an HTTP error
status of 500 - "Internal Server Error".
Using the server you just pointed out:
>>> import urllib2
>>> f = urllib2.urlopen("http://httpstatus.appspot.com/500")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
python2.5/urllib2.py", line 353, in _call_chain
result = func(*args)
python2.5/urllib2.py", line 499, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Internal Server Error
I couldn't figure out how to suppress that error message - urllib/
urllib2 is a confusing mess of a module - so I ended up writing to
the httplib module, which meant using some of the low-level modules.
Since I had to write my own (limited) MTOM parser, it also ended up
being easier with httplib than urllib2.
Because I don't have a full SOAP processor, I'm writing code to check
the response content-type and other fields in case a server upgrade
changes to another SOAP response type that I can't handle, or even
just changes the order of the items in the response.
It turns out, btw, that Tomcat doesn't report the correct content-
type, according to the MTOM spec.
> A more interesting test is what happens at the higher-level, i.e.
> with the actual payload/data that your code receives. At that level
> you can write unit tests against corner cases and see how your code
> reacts to various abnormal data sets.
I can, but that's my question. It seems to take a lot more work to
set up automated tests for those not-known-to-exist corner cases than
doing the manual tests that just check to see that the "raise
AssertionError('something bad happened')"-type code path doesn't
have a typo.
> A better test in this case would be to verify that the Excel
> document that you generate contains the data you expect.
Yeah, I haven't gotten to testing the Excel results yet, other than
visual inspection. How do I make sure that the text in column D
which contains more than 5 characters is rotated vertically?
There are ways to do it, and perhaps this is simply bad habits of
mine learned over long years of BASIC programming or some such
reason, but it feels easier to check the output visually than trying
to figure out some automated system system for this. In any case,
that's diverging from my question.
> Just verifying an exit code will not tell you anything terribly
> exciting about the correctness of your code.
Checking the error code has proved useful before:
- paths change and directories get reorganized. The exit code
tells me if the program didn't start.
- some programs use a time-based licenses (this one does) and if the
license fails then it might report the failure through an exit
As I found out today, this one doesn't work that way. :(
At least I know the failure mode now - it sends text to stdout
the does an exit(0).
- I don't know what the program does on all cases of invalid
input and the documentation isn't enough so I can't filter
out all bad input from the program. Some programs report
input errors with exit(error_code).
Is that part of code correctness? It feels like a pretty orthogonal
topic to me.
> I personally find mock testing extremely beneficial for simulating
> error conditions or exceptions that would otherwise be hard to
> intercept reliably (since they tend to be random). So you could
> have a mock Web service and simulate exceptions such as 'service
> not available' or timeouts. Then you can test how your code reacts
> to these exceptions.
> The other benefit of mock testing is to return known data (assuming
> the live Web service returns data that has values that vary wildly
> from one run to the other).
Perhaps I can rephrase my question. I know how to write mock systems
for all of the things I've talked about. the question I've been
thinking about as I write this code is the trade-off between:
- do I write code to test for cases that aren't known to occur,
based on experience that other similar code has done unexpected
things during rare circumstances?
- if so, do I test the code paths? Should I
- do manual tests that the code path executed suffice?
(The tests are always similar to:
raise AssertionError("message: %s" % value)
so there's very little that can go wrong with them.)
- or should I automate those tests?
- How much time should I spend setting up mock tests for
something that isn't known to ever occur?
I know that, for example, SQLite has a filesystem isolation layer
designed to test rare cases. I'm quite impressed with the code
coverage. But at least some of those rare tests are still for errors
that may occur and have occurred on some hardware. I'm talking about
for code where there's no evidence that a given failure mode will
dalke at dalkescientific.com
More information about the testing-in-python