[TIP] Defining a convention for running tests, #2

[Robert Collins]

On Sat, 2010-02-27 at 21:31 +0000, Michael Foord wrote:
> On 27/02/2010 20:55, Robert Collins wrote: 
> > On Sat, 2010-02-27 at 12:13 -0800, C. Titus Brown wrote:
> >   
> > > Hi all,
> > > 
> > > I've updated the README for my SomePackage example code,
> > >     
> > So, my key points are, I guess:
> > 
> > python -m somepackage.tests.run - isn't very useful:
> >  - doesn't define getting JUnitXML (is it a command line option? should
> > you use subunit? should you use nose-junit, should you use
> > xmltestrunner?)
> >   
> 
> It is no less useful than "setup.py test" and Titus intends it to be
> for a different situation - post installation where there is no
> setup.py. It provides a standard entry point for running tests. The
> issue of command line options is real but orthoganal.

It is no less useful, but its not more useful either. If we define API
conformance instead, then python -m <runner> somepackage.tests is no
less useful than *either* and can be more useful: runner glue can choose
result objects, provide command line handling that the user finds
atractive, provide parallel execution glue etc, while still being
standardised.


> So, how do we provide command line options to test running with our
> standard-mechanisms is a big question. For continuous integration
> (pony-build) standard output (unittest or nose run with verbose
> output) could be parsed and converted into XML. Not ideal but at least
> demonstrates that it *could* be done without *having* to standardise
> command line handling.

Think of the kittens!

> Whilst I agree with you (except debug is an abomination that should go
> away) 

Hey, we agree on that too :)

> I don't see how it is relevant here. Perhaps just a worthwhile
> observation supporting the load_tests protocol?

Yes, and also arguing that API support is more useful and powerful than
command line support.
...
> Well, it is a legacy protocol. If it goes into distutils2 then it will
> never go away. Test suites that want to be compatible with both can
> simply do:
> 
>     def load_tests(loader, tests, pattern):
>         return test_suite()

They can! If they have to change other things to use distutils2, then I
think documenting this is all that is needed.

Perhaps also, have distutils2 issing a Warning if it finds a test_suite
and no load_tests definition. (Having both is fine - backwards compat.
Having just test_suite probably indicates someone that has migrated to
distutils2 and not updated their test glue).

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20100228/9d1308d2/attachment-0001.pgp>