[TIP] Guidelines for where to put tests & how to package them
C. Titus Brown
ctb at msu.edu
Sat Feb 27 11:32:54 PST 2010
On Sat, Feb 27, 2010 at 07:27:34PM +0000, Michael Foord wrote:
> On 27/02/2010 19:15, C. Titus Brown wrote:
>> On Fri, Feb 26, 2010 at 07:36:59PM +0000, Michael Foord wrote:
>>>> test command performs test discovery (as a last recourse AFAICR, /me
>>>> cc-ing to P.J. Eby because probably he has further updates ;o) using a
>>>> loader: setuptools.comand.test.xxx
>>> I'd rather follow up with Tarek (cc'd) as the lead maintainer of both
>>> distribute (the maintained fork of setuptools) and of distutils(2).
>> Aieeee, I'm trying to come up with a sensible convention BEFORE we bring
>> it to the stdlib people ;).
> We have different goals in this discussion. You want a convention that
> can be used with existing versions of Python for pony-build, I am
> concerned with enshrining a good convention in the standard library. For
> what it is worth I think Tarek and I have arrived at a good solution for
> this (see below) with the painful caveat that I have to implement it for
> I *still* think that pony-build would be better served by adopting the
> standard library convention, and depending on unittest2 / distutils2 on
> the client to do that. :-) I probably won't mention this again...
No, you mistake me: pony-build (and experience with other CI systems) makes me
want to take it on, but I would like to have something sensible re testing in
the Hitchhiker's Guide to Packaging -- which defines packaging practice from
Python 2.4 onwards.
So where our goals differ is that you want to define a new convention
going forward (2.7+, 3.3+) and I want to document some convention that's
"good enough" & backwards compatible to 2.4.
>>>> so it would be very nice when defining packaging
>>>> conventions, layout ... not to enforce the use of that protocol, so
>>>> that if another discovery is used then it could still be compatible
>>>> with the more abstract good practices. (all this said from my selfish
>>>> position, I confess)
>>> Yes, but get_suite/test_suite is not supported, so distutils2 still
>>> *should* use a unittest supported protocol. Probably Tarek and I need to
>>> discuss how the setup.py test command should work for distutils2. It
>>> should support non-unittest tests as well (I guess?) so specifying a
>>> unittest suite is not necessarily ideal.
>> YES, please let's support non-unittest tests.
> distutils2 will have tests and testrunner arguments in the setup.py for
> specifying any testrunner (non-unittest or otherwise) and a test
> package. If you specify tests without a test runner then unittest will
> be used, if you specify neither then discovery will be attempted.
> With the load_tests protocol your test module or package (specified by
> tests) can return a unittest compatible suite even if tests are actually
> run under-the-hood with another framework. unittest will then only be
> used for reporting results.
OK, this still seems a bit restrictive; as we've discussed, unittest is
full of snakes that won't ever (or at least soon) be cleared out completely,
for the sake of backwards compatibility. So I'd like to avoid enshrining
load_tests as a convention.
> distutils2 won't have a runtime dependency on unittest2 - except if it
> isn't available then test discovery won't be attempted (you will have to
> supply explicit tests/testrunner arguments).
OK, sounds good.
C. Titus Brown, ctb at msu.edu
More information about the testing-in-python