[TIP] Model checking Python code
matthewharrison at gmail.com
Sat Sep 5 09:45:14 PDT 2009
Yeah, the grey areas. (Disclaimer: I'm not claiming to be an expert
in these matters)
On Sat, Sep 5, 2009 at 9:34 AM, Ned Batchelder<ned at nedbatchelder.com> wrote:
> Matt Harrison wrote:
> On Sat, Sep 5, 2009 at 6:17 AM, Ned Batchelder<ned at nedbatchelder.com> wrote:
> I'm still confused by the term "linear combinations of paths". What would a
> non-linear combination of them be? Why don't people just say, "combinations
> of paths"?
> Check out McCabe's papers. At a minimum they are worth it for the
> mugshot at the end. Here's the original, 33 years old now. It
> explains the theory. Here's a newer (a much longer) testing strategy
> one thats only about 13 years old. It gives more examples.
> I'm actually re-working that talk now to give at a local open source
> conference. Maybe I'll try and get the information out in a
> magazine article. Barring that I can put it on my blog.
> 0 -
> 1 - http://www.mccabe.com/iq_research_nist.htm
> 2 - http://2009.utosc.com/presentation/92/
> While we're on the theory of the matter, I'm also a bit troubled by the
> omission in these papers of exception throwing. Consider this C function:
> int foo(int a)
> return f1(a) + f2(a) + f3(a);
> It has a cyclomatic complexity of 1. Now look at a similar function in
> def foo(a):
> return f1(a) + f2(a) + f3(a)
> If any of f1, f2, or f3 can raise an exception (and in Python any one of
> them can), then doesn't this have a complexity of 4?
I would still say it has a complexity of 1. (Again this might be
grey, but it you want blacker and whiter, maybe Java is better where
you explicitly declare (most of) the exceptions thrown).
So, coverage for foo is simple. Now f1,f2 and f3 are public, so we'd
have tests for them too. And in those tests we'd cover common
exceptions (and explicitly put them in a try statement). I have have
code that can possibly divide by zero. When it divides by zero is
actually quite common, so it's in a try block. Those try blocks
create branches (which increase complexity), and so I should test for
those. Admittedly there's exceptions like "Out of Space", "Keyboard
Interrupt", that CAN appear that I don't test for at this level
(unless it's common, and then I'd explicitly catch it).
The question of when to move a might happen exception into a try block
is a good one. I'd like to see any discussion or pointers on this.
Again, I'm mostly concerned with the unit test level testing (at
first), and I think that you'd try to cover basic/common exceptions at
this level. I don't see them as adding too much confusion to
And if we're trying to
> measure paths executed, we're a bit stuck, because it might be that in fact
> f2 never raises an exception, so the "f2 raised" path will never be
> executed, and that's not a fault in the test suite, it's a lack of
> understanding of the system by the coverage tool.
We know that coverage isn't a panacea. I'd like it to indicate when
I've covered by (basis) paths though, or to indicate paths that I'm
missing, which line coverage can't do too well.
I recall that your preso at PyCon covered many grey areas. It would
be good to review them, though this might not be the right thread,
since Mark is probably annoyed that it is OT now.....
More information about the testing-in-python