[TIP] Guidelines for where to put tests & how to package them

Michael Foord fuzzyman at voidspace.org.uk
Tue Mar 2 15:14:49 PST 2010 

On 02/03/2010 19:21, holger krekel wrote:
> On Fri, Feb 26, 2010 at 21:53 +0000, Michael Foord wrote:
>    
>> On 25/02/2010 05:01, C. Titus Brown wrote:
>>      
>>> Hi all,
>>>
>>> here at PyCon there have been a lot of packaging discussions, so I thought
>>> I'd spend a bit of time outlining some suggestions for where to put
>>> tests and how to run them.  It's been a bit of a thorn in the side of
>>> (among other things) continuous integration systems that there's no
>>> standard way to run Python tests... so let's fix that!
>>>
>>> I've produced a simple draft proposal&   example where you put your unit tests
>>> under a package dir, somepackage/tests/.
>>>
>>> You can run these tests with
>>>
>>>     % python -m somepackage.tests.run
>>>
>>> and you can also do (if setuptools/distribute is installed)
>>>
>>>     % python setup.py test
>>>
>>>        
>> If you're happy to depend on distutils2 and unittest2 then very soon
>> there will be a standard "setup.py test" command that will continue to
>> work with future versions of Python. It will probably default to
>> unittest test discovery but allow you to provide a specific 'tests'
>> argument (either a module or suite) and a 'testrunner' (test_runner ?)
>> argument to specify a non-unittest runner if needed (testrunner defaults
>> to unittest.TextTestRunner, well actually unittest2.TextTestRunner).
>>      
> (i am on low bandwidth and vacation, just a quick note)
> I'd appreciate it if we had a more general mechanism
> than a unittest TestCase and a test runner.  Maybe just a callable
> with some directory argument that is to discover and run the tests?
> After all there are non-python tests and non-TestCase tests these
> days and introducing a standard should be general enough for that.
>    

There are three protocols suggested (and it would be nice to get that 
down to two... :-), none of which require tests to be TestCase based - 
although one of the protocols requires a test suite like object runnable 
with a TextTestRunner. As I explain below this is a very minimal 
requirement. The three protocols we have discussed are:

Packages provide a test sub-package with a run module that is able to 
collect and run the tests when executed with:

     python -m package.test.run

This obviously places no restriction on which test runner is used.

Under the auspices of distutils2, setup.py will gain a test command. It 
is possible (indeed likely) that this will take two optional parameters 
"test_suite" (or "tests" - still undecided) and "test_runner". The 
test_runner argument allows any runner, not limited to unittest.

If these arguments are not used, the suggestion is that "setup.py test" 
will do test discovery - using unittest. With the "load_tests" protocol 
(and this is what Robert has been suggesting ) the only requirement is 
that "load_tests" in any test module should return a suite-like-object 
capable of running tests and compatible with unittest TestResult. There 
is *no* requirement that tests be TestCase based or use any of the other 
unittest machinery. Even using py.test or nose under the hood it would 
be very simple for them to use the unittest protocols here as they are 
very simple.

The unittest protocol in practise just requires the following of a suite 
object:

* the suite should be callable (preferably with __call__ as an alias to 
a run method - although the TextTestRunner doesn't actually use that)
* __call__ should take a TestResult object - and should call the 
'appropriate' methods (addSuccess, addFailure, addError) for each test
* the test result expects individual test objects (passed into the add* 
methods) to have a shortDescription() method and for them to be str'able.
* the unittest protocol also suggests a countTestCases method (and 
useless horrible debug method), however the TextTestRunner doesn't use these

That's it... It is very easy for non-unittest test frameworks to create 
suite like objects that can be run by a unittest test runner, or even 
included in a suite alongside other unittest tests (TestSuite itself 
recurses into 'sub-suites' by calling them with the result object)...

All the best,

Michael Foord

> cheers&  thanks,
> holger
>
>
>    
>> This means that no *specific* layout for tests need to be mandated, so
>> long as it is compatible with test discovery or you do the test
>> collection yourself with the 'tests' argument.
>>
>> With this in place (which it isn't yet) it is unlikely that unittest /
>> distutils will adopt an official 'recommended' test layout, but don't
>> let that stop you... ;-)
>>
>> FWIW I would rather see ponybuild use distutils2 / unittest2 (i.e. the
>> recommended [1] community tools), but I understand your reluctance to
>> take a dependency on anything other than a single script on the client
>> side. A bootstrap script for installing dependencies would be one way
>> round this.
>>
>> All the best,
>>
>> Michael Foord
>>
>> [1] Recommended by me...
>>
>>      
>>> ---
>>>
>>> Full source at http://github.com/ctb/SomePackage or downloadable
>>> at
>>>
>>>      http://lyorn.idyll.org/~t/transfer/SomePackage.tar.gz
>>>
>>> Comments?  Thoughts?  Complaints?  Issues I missed?
>>>
>>> thanks,
>>> --titus
>>>
>>>        
>>
>> -- 
>> http://www.ironpythoninaction.com/
>> http://www.voidspace.org.uk/blog
>>
>> READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
>>
>>
>>
>> _______________________________________________
>> testing-in-python mailing list
>> testing-in-python at lists.idyll.org
>> http://lists.idyll.org/listinfo/testing-in-python
>>
>>      
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

More information about the testing-in-python mailing list