[TIP] Test coverage and parsing.

[Michał Kwiatkowski]

Hi again,

Sorry for late reply, it's a crazy start of the semester.

2009/10/6 Olemis Lang <olemis at gmail.com>:
> Considering what I've read about `QuickCheck` (I've discovered that
> I'm not an expert using Haskell :-/ ) it is able to generate test
> cases out of systems specs.

Yes, as long as those specs are expressed in Haskell.

> So I suppose that I'd need to implement
> one such model for SQL (isn't it ?) so that SQL statements be
> generated instead of unstructured data.

That's right, and you can build your generators on top of existing
ones, provided by QuickCheck, or in Python's case: PeckCheck. Some
built-in generators include things like a_boolean or an_int (Arbitrary
Bool and Arbitrary Integer respectively for Haskell). You can also
build new generators out of existing ones using combinator functions,
like a_list or a_choice (called listof and oneof in Haskell).

>  - Since I've found nothing about that, and you mentioned composable
>    operators and other features: Is it part of QuickCheck or is it an
>    extension? If so, which one ?
>  - Is it available too in the Py version?

If you look at the actual code for PeckCheck it's a little more than
200 lines. I think the idea is more important here than the actual
implementation. And the idea is this: Given that you have code written
in a functional style (i.e. without side effects) you can express
properties of the code in a mathematical-like notation using data
generators for doing actual checking. The produced test suite will be
closer to the textual specification and you can adjust the scale of
your tests more easily (by which I mean how many test cases for each
spec you want to run). It's kind of like fuzzy testing, but with a bit
less fuzz. ;-)

Now, the idea is simple, but you need a good framework to execute it.
Here's where all the versions differ. For example, Haskell leverages
its knowledge of types, making specifications less verbose.

> 1- It seems that CI environments are compulsory after adopting this
> approach. I mean, how could you know if test data is good enough to
> cover all relevant scenarios? What better tool than CI systems to
> measure such metrics ;o) ?

It's a general statement about all tests: you can't use them to prove
your code works, no matter how many of them you have. Still, IMHO
having 1000 diverse test cases auto-generated from a declarative spec
(and different set of 1000 each time!) is better than having a few
written manually. That's true for at least those programs that are
mostly pure (i.e. don't use side effects) and parsers and compilers
are a great example of that category.

In the particular case of the QuickCheck-like frameworks, right
distribution of generated values must be taken into consideration.
There are always some corner cases and you want them to be tested
reliably, not randomly (say in tenth of all test runs). That's another
area where a good framework can help the programmer express the
specifications right.

> 2- This approach and these libs are both very useful. Shouldn't we
> build at least a few of them and enhance support for this in CI tools
> ?

I agree. Specs for things like XML, SQL or JSON are there, but having
executable specs would be even better. :-) Seriously though, right now
it sounds like a pipe dream to have full executable specs, but we (as
a more general "development community") could do better.

One example of that, from the Java world, is the Technology
Compatibility Kit - a comprehensive test suite used for checking
compatibility of different JVM implementations.

> I like it ! Damn it ! I couldn't sleep last night :P
>
> Thnx very, very, very much !

The please is mine. It's always nice to meet another test infected person. :-)

Cheers,
mk