[TIP] Guidelines for where to put tests & how to package them

[C. Titus Brown]

On Sat, Feb 27, 2010 at 07:49:54PM +0000, Michael Foord wrote:
> On 27/02/2010 19:39, C. Titus Brown wrote:
>> On Fri, Feb 26, 2010 at 09:53:41PM +0000, Michael Foord wrote:
>>    
>>> On 25/02/2010 05:01, C. Titus Brown wrote:
>>>      
>>>> Hi all,
>>>>
>>>> here at PyCon there have been a lot of packaging discussions, so I thought
>>>> I'd spend a bit of time outlining some suggestions for where to put
>>>> tests and how to run them.  It's been a bit of a thorn in the side of
>>>> (among other things) continuous integration systems that there's no
>>>> standard way to run Python tests... so let's fix that!
>>>>
>>>> I've produced a simple draft proposal&   example where you put your unit tests
>>>> under a package dir, somepackage/tests/.
>>>>
>>>> You can run these tests with
>>>>
>>>>     % python -m somepackage.tests.run
>>>>
>>>> and you can also do (if setuptools/distribute is installed)
>>>>
>>>>     % python setup.py test
>>>>        
>>> If you're happy to depend on distutils2 and unittest2 then very soon
>>> there will be a standard "setup.py test" command that will continue to
>>> work with future versions of Python. It will probably default to
>>> unittest test discovery but allow you to provide a specific 'tests'
>>> argument (either a module or suite) and a 'testrunner' (test_runner ?)
>>> argument to specify a non-unittest runner if needed (testrunner defaults
>>> to unittest.TextTestRunner, well actually unittest2.TextTestRunner).
>>>      
>> Yes, but it doesn't specify how to run the tests in situations where
>> setup.py isn't available (e.g. post-install).  The distro packaging folk
>> like that idea, as do I.
>
> Ok - that's a good point that I'd forgotten about. Worth thinking about  
> some more then. The danger is that we end up with several different  
> conventions for different use cases. The corresponding danger is that we  
> end up with a sub-optimal use case (additional complication in the form  
> of the run module and mandating package layout) for developers.

Right.

> Perhaps the convention could be that you provide an executable run  
> module only if the distutils2 defaults aren't adequate. In this case the  
> arguments you supply in setup.py tests/testrunner arguments will have to  
> do the same job as your run module (duplication).

I like the idea: default to something sensible and reasonably simple, but
allow for complications.

> We could come up with a more complex scheme for distutils2 that meets  
> all these usecases:
>
> If no arguments are specified in setup.py then "setup.py test" does the  
> following:
>
> For each package in the distribution that supplies a test sub-package  
> including a run module, execute the equivalent of:
>
> python -m package.test.run
>
> For all packages that don't provide a test.run sub-package and module  
> attempt test discovery.
>
> If tests/testrunner arguments are supplied to setup.py then those will  
> be used and the programmer should also provide a test.run if he wants  
> tests to be able to be run without setup.py being used.
>
> This kind of makes the setup.py arguments redundant, or at least  
> slightly harmful to use them without providing test.run as well.

I don't entirely understand this but I need to go educate myself,
I think!

cheers,
--titus
-- 
C. Titus Brown, ctb at msu.edu