[TIP] Guidelines for where to put tests & how to package them

[C. Titus Brown]

On Sat, Feb 27, 2010 at 07:27:34PM +0000, Michael Foord wrote:
> On 27/02/2010 19:15, C. Titus Brown wrote:
>> On Fri, Feb 26, 2010 at 07:36:59PM +0000, Michael Foord wrote:
>>    
>>>> test command performs test discovery (as a last recourse AFAICR, /me
>>>> cc-ing to P.J. Eby because probably he has further updates ;o) using a
>>>> loader: setuptools.comand.test.xxx
>>>>
>>>>        
>>> I'd rather follow up with Tarek (cc'd) as the lead maintainer of both
>>> distribute (the maintained fork of setuptools) and of distutils(2).
>>>      
>> Aieeee, I'm trying to come up with a sensible convention BEFORE we bring
>> it to the stdlib people ;).
>
> We have different goals in this discussion. You want a convention that  
> can be used with existing versions of Python for pony-build, I am  
> concerned with enshrining a good convention in the standard library. For  
> what it is worth I think Tarek and I have arrived at a good solution for  
> this (see below) with the painful caveat that I have to implement it for  
> distutils2.
>
> I *still* think that pony-build would be better served by adopting the  
> standard library convention, and depending on unittest2 / distutils2 on  
> the client to do that. :-) I probably won't mention this again...

No, you mistake me: pony-build (and experience with other CI systems) makes me
want to take it on, but I would like to have something sensible re testing in
the Hitchhiker's Guide to Packaging -- which defines packaging practice from
Python 2.4 onwards.

So where our goals differ is that you want to define a new convention
going forward (2.7+, 3.3+) and I want to document some convention that's
"good enough" & backwards compatible to 2.4.

>>>> so it would be very nice when defining packaging
>>>> conventions, layout ... not to enforce the use of that protocol, so
>>>> that if another discovery is used then it could still be compatible
>>>> with the more abstract good practices. (all this said from my selfish
>>>> position, I confess)
>>>>        
>>> Yes, but get_suite/test_suite is not supported, so distutils2 still
>>> *should* use a unittest supported protocol. Probably Tarek and I need to
>>> discuss how the setup.py test command should work for distutils2. It
>>> should support non-unittest tests as well (I guess?) so specifying a
>>> unittest suite is not necessarily ideal.
>>>      
>> YES, please let's support non-unittest tests.
>>    
> distutils2 will have tests and testrunner arguments in the setup.py for  
> specifying any testrunner (non-unittest or otherwise) and a test  
> package. If you specify tests without a test runner then unittest will  
> be used, if you specify neither then discovery will be attempted.

Great!

> With the load_tests protocol your test module or package (specified by  
> tests) can return a unittest compatible suite even if tests are actually  
> run under-the-hood with another framework. unittest will then only be  
> used for reporting results.

OK, this still seems a bit restrictive; as we've discussed, unittest is
full of snakes that won't ever (or at least soon) be cleared out completely,
for the sake of backwards compatibility.  So I'd like to avoid enshrining
load_tests as a convention.

> distutils2 won't have a runtime dependency on unittest2 - except if it  
> isn't available then test discovery won't be attempted (you will have to  
> supply explicit tests/testrunner arguments).

OK, sounds good.

cheers,
--titus
-- 
C. Titus Brown, ctb at msu.edu