[TIP] Calculating coverage of runtime-generated functions

Ned Batchelder ned at nedbatchelder.com
Sat May 2 17:12:15 PDT 2009

Nicolas Trangez wrote:
> Hey Ned, all,
> On 02 May 2009, at 23:24, Ned Batchelder wrote:
>> My initial impression is that you don't have two functions here, at
>> least not as far as the source is concerned.  You're in the same
>> position as if your code were:
>> def fun(n, a):
>>    return n * a
>> class DemoTest(unittest.TestCase):
>>    def test_two(self):
>>        self.assertEquals(fun(2, 2), 4)
>> There are cases in your code you haven't tested (the case where n is 3)
>> either way.
> Of course. The code I provided was just an example, where the amount 
> of possible generated functions is endless, were the actual scenario 
> has a finite number of generated functions.
Infinite values vs. finite values is a minor point: the problem is still 
that you want more than statement coverage, you want value coverage 
(measurement of which values have been covered).  None of the Python 
coverage tools provide this feature.
> Hopes all of this makes some sense...
Yes, it does, but it doesn't change my view of the problem.  You have 
code where every statement is executed, but you know that there are 
interesting tests that haven't been executed.  That's because you 
understand that the important states of the program include not just the 
program counter (the statements that have been executed), but also the 
values of the variables.

It would be wonderful to have a tool that could help measure value 
coverage, but Coverage is not it.  If you have ideas about how to 
automate value coverage measurement, I'd love to hear them.  I've given 
it no thought, but it could be very valuable.


Ned Batchelder, http://nedbatchelder.com

More information about the testing-in-python mailing list