[TIP] LNCS: Accuracy of test cases generated by pythoscope WAS: Test coverage and parsing.
olemis at gmail.com
Mon Oct 12 09:14:21 PDT 2009
@Ryan : I'm posting your message to the list. Hope you don't mind ...
On Mon, Oct 12, 2009 at 10:32 AM, Ryan Freckleton
<ryan.freckleton at gmail.com> wrote:
> 2009/10/12 Olemis Lang <olemis at gmail.com>:
>> And here is where the meteorite falls ! ( especially because maybe I
>> don't fully understand how things work ;o)
>> `pythoscope` may be considered as an MB test generator that considers
>> as input the (legacy) code itself (... or an equivalent representation
>> like ASTs ;o) and employs that model to generate test cases. If this
>> is correct (CMIIW) then users are facing the situation mentioned
>> above: «tests and the implementation are not sufficiently independent»
>> and «any errors in that model (i.e. legacy code) will generate
>> incorrect implementation code (i.e. they are both the same thing), but
>> the tests will contain the same errors, so no test failures will be
>> So ...
>> - Is it possible that something like that be happening
>> if using `pythoscope` ?
>> - If true ... Is there any way (provided by pythoscope or not ;o)
>> to overcome this ?
> Looking on the wikipedia page for Model-Based-Testing, it looks like
> the test cases are created from a model of the system's desired
Well it *SHOULD* be created from a model of the system's desired
behavior; but sometimes that's not what happens, and at some extent
this is the case ... ;-/
The system's desired behavior may not be the softw real behavior.
Otherwise why are we testing it, anyway ?
> Assuming that your legacy code is perfect (hah!) then
> pythoscope will generate executable tests that match this model.
That's the good one ! You'll receive tests that check that your legacy
code does exactly what the code is executing. I mean the softw does
the following :
- Builds a model of the computations performed by target softw
- Infers some rules and conditions that *SHOULD* be met, but
using the same code
- Generates test cases to check that the conditions inferred are
Isn't it obvious that tests should always pass ? (pls, CMIIW ... this
is just my naive idea ... ) Besides if tests fail then the only thing
yo can ensure is that the test generator has failed or (if newer or
modified versions of legacy are in use ;o) that the new system behaves
somehow different ...
> In either case unit tests are still very valuable as developer tools
> and bug detectors, even if you don't assume that your code is perfect.
> They allow refactoring, and the detection of bugs within single units
> of code.
... just like in this case, but not that the outcomes are more or less accurate
The workflow I envisioned to use `pythoscope` was to follow the
aforementioned steps but, once tests are generated, review all the
test cases in order to confirm that the assertions in there are really
correct. Otherwise, a bug was found and thus the test case has to be
modified somehow in order to meet the reqs.
BTW this is much better than manual TCs
> The remaining question is "how do you verify that your legacy code
> matches your model of desired behavior?" There isn't a way to do this
> automatically :(, you have to determine what's valuable to your
> stakeholders and develop tests that will give you information about
> whether those valuable characteristics are present in the software.
Ahhh ! There we have it .
My conclusions : `pythoscope` is very useful, but better take care of
what you'r doing ;o)
- At what extent `pythoscope` helps with test coverage ?
Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/
More information about the testing-in-python