[TIP] hackish generative tests (was: Re: Generating tests, II (was: Re: Meta-test methods...))

[holger krekel]

Hi Doug, 

i guess i am somewhat responsible for the idea of "generative
tests" and maybe you find it suprising to hear: i mostly 
agree with what you say.  Tests generators as they stand 
are somewhat hackish.  Some more inline comments ... 

On Fri, Apr 24, 2009 at 15:35 -0400, Douglas Philips wrote:
> On or about 2009 Apr 24, at 3:10 PM, C. Titus Brown indited:
> 
> > I'm trying to figure out why I dislike the non-decorator tagging so  
> > much; I think because I want
> > to know what the signature of a test function is up front, without
> > looking at the end.  That conflicts with 'yield' too, though.  I guess
> > I'm just inconsistent.
> 
> Cracking open a method and examining how it is implemented is just  
> plain and simple: a hack. If it were in code I was reviewing, I would  
> classify it as a bug. If there were no other way, sure, it might be  
> tolerable. But there are other ways. At least two that that are  
> actually Pythonic.

"cracking open a method" i don't consider a fitting term 
for checking if a function is a generator. 

In any case, introspection and reflection is something i consider pythonic 
so if you want to say that using such is a hack in general, i disagree. 
Testing is an application of meta-programming and meta-programming 
naturally involves reflection techniques. 

however, ... 

> If the py.test/nose dudes had used new naming convention, would you  
> really have balked at it? Would you have thought it complicated or un- 
> usable? Would you have thought "Oh, that makes sense, this is a  
> different part of the testing eco-system" of course it needs a  
> different name?

... this is a good point.  Dual-using the "test_*" naming 
convention for test generators i now consider a bad idea that 
lead to implementation complications both for py.test and nose.  
In that sense i agree to your "cracking open a method" critcism. 

FYI i also agree to some of the critical viewpoints on generative tests 
expressed here: 
http://bruynooghe.blogspot.com/2008/09/generative-tests.html

However, I think that instrumenting a test run to invoke tests
with multiple different parameters or in multiple environments
is useful and worthwhile.  

So i'd like to phase out (probably making it an 
optional plugin) yield-mediated generative tests once 
i implemented a mechanism based on 
http://codespeak.net/py/trunk/test/funcargs.html 
that allows to invoke a test function repeatedly 
with multiple parameter sets.  

cheers,
holger