[TIP] Generating tests dynamically (was Re: Meta-test methods...)

[Michael Foord]

C. Titus Brown wrote:
> On Fri, Apr 24, 2009 at 03:25:33PM +1000, Robert Collins wrote:
> -> On Fri, 2009-04-24 at 00:01 -0400, Douglas Philips wrote:
> -> > Anyway, I double-dare you pay attention to the issue at hand. :)
> -> > I never said generators were bad. I said that checking a test method  
> -> > for being a generator was a hack, seeminly cool at first, but upon  
> -> > further thought it is just as a proof of concept that it is useful
> -> > to  
> -> > have a way to dynamically generate a sequence of test
> -> > methods/functions.
> -> > Testing only for generators is both opaque and, critically, fails to  
> -> > generalize to arbitrary python sequences.
> -> 
> -> At least in unittest, you can do this just by generating whatever tests
> -> you want in TestSuite.__iter__.
> -> 
> -> countTestCases clearly can't do a lot there, but that is the only method
> -> in the protocol that conflicts with generation.
>
> This may seem tangential but IMO is not: could someone document the
> !%#!$!# out of unittest, please?
>
> Reading through this
>
> 	http://docs.python.org/library/unittest.html
>
> doesn't give me any clue as to how to extend TestSuite to do this; sure,
> I can read (and modify) the source code, but basing a test suite for
> multiple versions of Python on unittest internals stirkes me as a tad
> risky.
>   

That's one of the difficulties of unittest - the extension mechanisms 
are unclear. Once you do read the source code it isn't so hard to work 
out - but there is no real guidance as to what the intended (or 
approved) extension points are and which are implementation details.

I guess the situation is that it wasn't really considered, and so people 
have extended unittest in just about every possible way and we now have 
to assume that people are now relying on all the private APIs.

Some of the extension mechanisms that Robert has suggested (adding extra 
method calls in runTests) may make this easier.

I'd like to write up how I've extended unittest and maybe address some 
of these issues once the current batch of changes are out of the way.

Michael Foord


> With nose, I'm fairly sure that once given an API for generating tests
> in my test suite, that API will continue to work.  That's partly because
> I know (roughly) where Jason Pellerin lives, and partly because the API
> is rather minimal and it can be implemented in multiple ways.  With
> unittest, I don't have a good sense for what guarantees there are, other
> than that what is specified in the docs.  I consider everything else to
> be internal & subject to change and resulting backwards incompatibility.
>
> cheers,
> --titus
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog