[TIP] Calculating coverage of runtime-generated functions
Nicolas.Trangez at Sun.COM
Sat May 2 15:45:21 PDT 2009
Hey Ned, all,
On 02 May 2009, at 23:24, Ned Batchelder wrote:
> My initial impression is that you don't have two functions here, at
> least not as far as the source is concerned. You're in the same
> position as if your code were:
> def fun(n, a):
> return n * a
> class DemoTest(unittest.TestCase):
> def test_two(self):
> self.assertEquals(fun(2, 2), 4)
> There are cases in your code you haven't tested (the case where n is
> either way.
Of course. The code I provided was just an example, where the amount
of possible generated functions is endless, were the actual scenario
has a finite number of generated functions.
> My point is that the source lines are being covered in both your code
> and mine, but there are cases not covered because the actual values
> in the functions matter.
Indeed, but this is not the case in the actual code this applies to,
> I don't know how to deal with it though: how
> should coverage decide what values are sufficient to cover the cases?
Good point. This is why, IMHO, not every case (every possible
generated function) should be checked for coverage, but every case
which is *exposed* as an API.
Here's the actual case: we got some sort of library/framework/...
which exposes an API which is not supposed to be a developer-oriented
API, but more something like a very basic scripting thing.
Part of the library exposes hashing functions, which allows a library
user to get the hash value of a string, a file or a stream using
several algorithms (md5, sha1, sha256, crc32,...). Since the library
also contains some sort of interactive shell, having eg a 'hashString'
function which takes a string and some hash-function specifier is not
really desired, something like 'hashStringSHA1' and 'hashStringMD5' is.
Because all this uses hashlib which exposes a single interface, I
created a function generator to create a string, file and fd hash
method for every hash function we support.
Implementation is at , testcase at . The code is currently
pretty well covered, but that's only because I *know* it is: as long
as all CRC32 and (eg) MD5 functions are tested, any current coverage
tool would tell me there's 100% coverage, which is not true from a
functionality perspective (it is from lines-of-code perspective,
A similar case would be code where several classes inherit from one
single base class, and have one more (class) attribute, where this
base class has one method which requires/uses this attribute to do
it's job (consider the base class to be static). This is what's +-
done at an extremely basic level in the test source .
As long I create one testcase which tests this method on one child
class, the file will appear to be 100% covered, since the attribute
definition on the childs will be visited during compilation and the
code implementing the static function is covered, but all
'implementations' (whose number is the number of child classes) are
not covered at all, which can be an issue if the values stored in this
attribute are non-obvious.
Hopes all of this makes some sense...
> Nicolas Trangez wrote:
>> In a project I'm working on there are several constructs like the one
>> pasted at the end of this email. Recently we wanted to enhance the
>> test coverage of the project, and measure this as well, using a
>> standard code coverage tool.
>> I've been looking at both coverage.py (3.0 beta) and figleaf, both
>> integrated in nose, which is the test runner we're using, but none of
>> them cover the problem, not to my surprise since it seems rather hard
>> to tackle. Given the code snippet below (which is obviously an
>> oversimplification of the real code), coverage tools report 100%
>> coverage, whilst function 'three' is not tested at all. If the
>> generated functions would use more complex components in their
>> closure, one really wants to test all of them.
>> The main question is: is there any existing tool which allows to
>> code coverage of code for which no 1-to-1 mapping to sourcecode is
>> If not I guess it'd be useful to work on this, but both from a
>> technical as well as from a UI point of view I expect some
>> Any ideas are welcome :-)
>> def gen(n):
>> def fun(a):
>> return n * a
>> return fun
>> two = gen(2)
>> three = gen(3)
>> import unittest
>> class DemoTest(unittest.TestCase):
>> def test_two(self):
>> self.assertEquals(two(2), 4)
>> testing-in-python mailing list
>> testing-in-python at lists.idyll.org
> Ned Batchelder, http://nedbatchelder.com
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
More information about the testing-in-python