[TIP] Guidelines for where to put tests & how to package them

Michael Foord fuzzyman at voidspace.org.uk
Sat Feb 27 11:49:54 PST 2010

On 27/02/2010 19:39, C. Titus Brown wrote:
> On Fri, Feb 26, 2010 at 09:53:41PM +0000, Michael Foord wrote:
>> On 25/02/2010 05:01, C. Titus Brown wrote:
>>> Hi all,
>>> here at PyCon there have been a lot of packaging discussions, so I thought
>>> I'd spend a bit of time outlining some suggestions for where to put
>>> tests and how to run them.  It's been a bit of a thorn in the side of
>>> (among other things) continuous integration systems that there's no
>>> standard way to run Python tests... so let's fix that!
>>> I've produced a simple draft proposal&   example where you put your unit tests
>>> under a package dir, somepackage/tests/.
>>> You can run these tests with
>>>     % python -m somepackage.tests.run
>>> and you can also do (if setuptools/distribute is installed)
>>>     % python setup.py test
>> If you're happy to depend on distutils2 and unittest2 then very soon
>> there will be a standard "setup.py test" command that will continue to
>> work with future versions of Python. It will probably default to
>> unittest test discovery but allow you to provide a specific 'tests'
>> argument (either a module or suite) and a 'testrunner' (test_runner ?)
>> argument to specify a non-unittest runner if needed (testrunner defaults
>> to unittest.TextTestRunner, well actually unittest2.TextTestRunner).
> Yes, but it doesn't specify how to run the tests in situations where
> setup.py isn't available (e.g. post-install).  The distro packaging folk
> like that idea, as do I.

Ok - that's a good point that I'd forgotten about. Worth thinking about 
some more then. The danger is that we end up with several different 
conventions for different use cases. The corresponding danger is that we 
end up with a sub-optimal use case (additional complication in the form 
of the run module and mandating package layout) for developers.

Perhaps the convention could be that you provide an executable run 
module only if the distutils2 defaults aren't adequate. In this case the 
arguments you supply in setup.py tests/testrunner arguments will have to 
do the same job as your run module (duplication).

We could come up with a more complex scheme for distutils2 that meets 
all these usecases:

If no arguments are specified in setup.py then "setup.py test" does the 

For each package in the distribution that supplies a test sub-package 
including a run module, execute the equivalent of:

python -m package.test.run

For all packages that don't provide a test.run sub-package and module 
attempt test discovery.

If tests/testrunner arguments are supplied to setup.py then those will 
be used and the programmer should also provide a test.run if he wants 
tests to be able to be run without setup.py being used.

This kind of makes the setup.py arguments redundant, or at least 
slightly harmful to use them without providing test.run as well.


>> FWIW I would rather see ponybuild use distutils2 / unittest2 (i.e. the
>> recommended [1] community tools), but I understand your reluctance to
>> take a dependency on anything other than a single script on the client
>> side. A bootstrap script for installing dependencies would be one way
>> round this.
> A solution that involves complicating things is not so good; one reason
> why the stdlib is so awesome is that it's always there!
> :)
> cheers,
> --titus


READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

More information about the testing-in-python mailing list