[TIP] new test parametrization / deprecating "yield"

[holger krekel]

On Fri, May 15, 2009 at 01:57 -0400, Douglas Philips wrote:
> On or about 2009 May 14, at 8:08 AM, holger krekel indited:
>> On Wed, May 13, 2009 at 15:27 -0400, Doug Philips wrote:
>>> I am not clear from that post and the py.test funcargs page
>>> it links to, just where/how many pytest_generate_tests
>>> functions there can be and when in the test-running-timeline
>>> they/it are run.
>>
>> I see.  I've written up a new blog post which hopefully clarifies
>> this and explains how the generate-tests hook can live
>> in a "local" or "global" plugin:
>>
>>    http://tetamap.wordpress.com/2009/05/14/putting-test-hooks
>
> Thanks. I'm still not clear on when the funcarg providers are called.
> Are they called as the modules are being loaded? (that doesn't seem  
> right).
> Are they called "just in time" as the test method itself is about to be 
> called?

OK, let me try to paraphrase the full funcarg-related protocol in detail: 

1. pytest_generate_tests hooks are called when modules
   are being loaded ("during collection").  This is different from 
   the actual setup and calling of a test function. 

2. Just-in-Time before running a function it is setup which includes
   having values for all function arguments: 

   a) The arguments that were directly provided through 
      metafunc.addcall(funcargs=...) will be taken as is.  

   b) Each missing function argument will be looked up by
      discovering a pytest_funcarg__${ARGNAME} function. 
      This provider function receives a request object, see
      http://codespeak.net/py/trunk/test/funcargs.html#request-object

Here is how you can have setup function arguments "just-in-time". 

    def pytest_generate_tests(metafunc):
        metafunc.addcall(param=1)
        metafunc.addcall(param=2)

    def pytest_funcarg__ARGNAME(request):
        return ExpensiveSetup(request.param)

We generate two test function calls.  The ARGNAME provider 
will be called twice, with the parameter 1 and 2 respectively. 

Briefly speaking, generate_tests passes a "param" when adding
a call that a "just-in-time" funcarg provider can see on its request
and use to parametrize expensive setup.

You can of course cache the setup of function arguments
and finalize/teardown e.g. on test process teardown. 
This means that you do not need any setup/teardown per
method, class etc. and can implement all setup/teardown
logic in exactly one place. 

Makes sense?  
If so i'd add something like the above to the docs or
to another blog post. 

cheers,
holger

> --
>
> This section also caught my eye:
> multiple pytest_generate_hook implementations
> ...
> So, you say, what about a test function with multiple arguments ? could 
> each function argument come from a different generating provider?
>
> This would mean that multiple generators act independently but want to  
> collaborate and combine their values for a given test functions. Well,  
> if you encounter a real need for it, please come forward and we?ll think 
> up a fitting API extension. A couple of days ago i had a "combining 
> funcargs" API call implemented but decided to remove it because i try 
> hard these days to only add features that have a proven need.
> ...
>
> I had thought combining values would be a natural extension.
> For example, one of the things that -some- of our device tests needs is a 
> power controller.
> But not all test code needs it.
> Right now our common TestCase base class arranges for setUp to stash the 
> power controller onto 'self' (during setUp). So that any test which wants 
> a power controller will have it. IMHO it would be a lot cleaner if the 
> test method simply declared it funcargs style:
>
> def test_rapid_fire_power_cycling(self, power_controller, ...):
>     ...
>     while <...>:
>         power_controller.off()
>         sleep(self.power_off_delay
> 	power_controller.on()
>
> the power_controller func_arg being an independent entity for other  
> funcarg supplied parameters.
>
> But again I'm not doing unit testing, I'm just using the unittest  
> framework to do device testing. :)
>
> --Doug
>

-- 
Metaprogramming, Python, Testing: http://tetamap.wordpress.com
Python, PyPy, pytest contracting: http://merlinux.eu