[TIP] Calculating coverage of runtime-generated functions

[Nicolas Trangez]

On Sat, 2009-05-02 at 20:12 -0400, Ned Batchelder wrote:
> Nicolas Trangez wrote:
> > Hey Ned, all,
> >
> > On 02 May 2009, at 23:24, Ned Batchelder wrote:
> >
> >> My initial impression is that you don't have two functions here, at
> >> least not as far as the source is concerned.  You're in the same
> >> position as if your code were:
> >>
> >> def fun(n, a):
> >>    return n * a
> >>
> >> class DemoTest(unittest.TestCase):
> >>    def test_two(self):
> >>        self.assertEquals(fun(2, 2), 4)
> >>
> >> There are cases in your code you haven't tested (the case where n is 3)
> >> either way.
> >
> > Of course. The code I provided was just an example, where the amount 
> > of possible generated functions is endless, were the actual scenario 
> > has a finite number of generated functions.
> Infinite values vs. finite values is a minor point: the problem is still 
> that you want more than statement coverage, you want value coverage 
> (measurement of which values have been covered).  None of the Python 
> coverage tools provide this feature.

Which is what I thought (especially after reading coverage.py and
figleaf sources).

The thing is, although I do agree from a VM point of view this is value
coverage, from a functionality POV this value is fixed so it's
essentially a new function as if you'd write it manually
(functools.partial is a similar case), where some of the code uses a
value not stored on the stack of the function itself but in some closure
cell containing a constant value.

This is some unknown construct in 'common' static-typed languages where
statement coverage ~= function coverage.

> >
> > Hopes all of this makes some sense...
> >
> Yes, it does, but it doesn't change my view of the problem.  You have 
> code where every statement is executed, but you know that there are 
> interesting tests that haven't been executed.  That's because you 
> understand that the important states of the program include not just the 
> program counter (the statements that have been executed), but also the 
> values of the variables.

Constant, non-parameter, non-global values.

> 
> It would be wonderful to have a tool that could help measure value 
> coverage, but Coverage is not it.

I know.

>   If you have ideas about how to 
> automate value coverage measurement, I'd love to hear them.  I've given 
> it no thought, but it could be very valuable.

I don't like to call this value coverage since that's a much more
general thing. I do have some ideas but those involve source code
changes (decorations on the applicable functions) and changes in both
test runners as well as coverage tools. Not sure it's possible at all,
and even if it is it's rather ugly. Will try to get something working
though.

Nicolas