[TIP] Calculating coverage of runtime-generated functions

[Nicolas Trangez]

Hey Ned, all,

On 02 May 2009, at 23:24, Ned Batchelder wrote:

> My initial impression is that you don't have two functions here, at
> least not as far as the source is concerned.  You're in the same
> position as if your code were:
>
> def fun(n, a):
>    return n * a
>
> class DemoTest(unittest.TestCase):
>    def test_two(self):
>        self.assertEquals(fun(2, 2), 4)
>
> There are cases in your code you haven't tested (the case where n is  
> 3)
> either way.

Of course. The code I provided was just an example, where the amount  
of possible generated functions is endless, were the actual scenario  
has a finite number of generated functions.

> My point is that the source lines are being covered in both your code
> and mine, but there are cases not covered because the actual values  
> used
> in the functions matter.

Indeed, but this is not the case in the actual code this applies to,  
see further.

>  I don't know how to deal with it though: how
> should coverage decide what values are sufficient to cover the cases?

Good point. This is why, IMHO, not every case (every possible  
generated function) should be checked for coverage, but every case  
which is *exposed* as an API.

Here's the actual case: we got some sort of library/framework/...  
which exposes an API which is not supposed to be a developer-oriented  
API, but more something like a very basic scripting thing.

Part of the library exposes hashing functions, which allows a library  
user to get the hash value of a string, a file or a stream using  
several algorithms (md5, sha1, sha256, crc32,...). Since the library  
also contains some sort of interactive shell, having eg a 'hashString'  
function which takes a string and some hash-function specifier is not  
really desired, something like 'hashStringSHA1' and 'hashStringMD5' is.
Because all this uses hashlib which exposes a single interface, I  
created a function generator to create a string, file and fd hash  
method for every hash function we support.
Implementation is at [1], testcase at [2]. The code is currently  
pretty well covered, but that's only because I *know* it is: as long  
as all CRC32 and (eg) MD5 functions are tested, any current coverage  
tool would tell me there's 100% coverage, which is not true from a  
functionality perspective (it is from  lines-of-code perspective,  
though).

A similar case would be code where several classes inherit from one  
single base class, and have one more (class) attribute, where this  
base class has one method which requires/uses this attribute to do  
it's job (consider the base class to be static). This is what's +-  
done at an extremely basic level in the test source [2].

As long I create one testcase which tests this method on one child  
class, the file will appear to be 100% covered, since the attribute  
definition on the childs will be visited during compilation and the  
code implementing the static function is covered, but all  
'implementations' (whose number is the number of child classes) are  
not covered at all, which can be an issue if the values stored in this  
attribute are non-obvious.


Hopes all of this makes some sense...

Nicolas

[1] http://staging.pymonkey.org/trac/browser/packages/pymonkey/core/hash.py
[2] http://staging.pymonkey.org/trac/browser/test/test_hash.py

>
>
> --Ned.
> http://nedbatchelder.com
>
> Nicolas Trangez wrote:
>> Hija,
>>
>> In a project I'm working on there are several constructs like the one
>> pasted at the end of this email. Recently we wanted to enhance the
>> test coverage of the project, and measure this as well, using a
>> standard code coverage tool.
>>
>> I've been looking at both coverage.py (3.0 beta) and figleaf, both
>> integrated in nose, which is the test runner we're using, but none of
>> them cover the problem, not to my surprise since it seems rather hard
>> to tackle. Given the code snippet below (which is obviously an
>> oversimplification of the real code), coverage tools report 100%
>> coverage, whilst function 'three' is not tested at all. If the
>> generated functions would use more complex components in their
>> closure, one really wants to test all of them.
>>
>> The main question is: is there any existing tool which allows to  
>> check
>> code coverage of code for which no 1-to-1 mapping to sourcecode is
>> available?
>>
>> If not I guess it'd be useful to work on this, but both from a
>> technical as well as from a UI point of view I expect some  
>> challenges.
>> Any ideas are welcome :-)
>>
>> Thanks,
>>
>> Nicolas
>>
>>
>> def gen(n):
>>     def fun(a):
>>         return n * a
>>     return fun
>>
>> two = gen(2)
>> three = gen(3)
>>
>> import unittest
>> class DemoTest(unittest.TestCase):
>>     def test_two(self):
>>         self.assertEquals(two(2), 4)
>>
>> _______________________________________________
>> testing-in-python mailing list
>> testing-in-python at lists.idyll.org
>> http://lists.idyll.org/listinfo/testing-in-python
>>
>>
>
> -- 
> Ned Batchelder, http://nedbatchelder.com
>
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python