[TIP] Defining a convention for running tests, #2

[Michael Foord]

On 27/02/2010 20:55, Robert Collins wrote:
> On Sat, 2010-02-27 at 12:13 -0800, C. Titus Brown wrote:
>    
>> Hi all,
>>
>> I've updated the README for my SomePackage example code,
>>      
> So, my key points are, I guess:
>
> python -m somepackage.tests.run - isn't very useful:
>   - doesn't define getting JUnitXML (is it a command line option? should
> you use subunit? should you use nose-junit, should you use
> xmltestrunner?)
>    

It is no less useful than "setup.py test" and Titus intends it to be for 
a different situation - post installation where there is no setup.py. It 
provides a standard entry point for running tests. The issue of command 
line options is real but orthoganal.


>   - doesn't define command line parameters except implicitly as 'the
> unittest ones' based on your sample code; in which case aiming for API
> conformance is more useful.
>
> These concerns also apply to setup.py test. As you say '
>   - Titus Brown: I also don't really understand what the arguments to
>     'setup.py test' are.'
> For integration with CI tools, the primary, defacto method is 'get a
> JUnit XML file on disk', with everything else secondary.
>    

So, how do we provide command line options to test running with our 
standard-mechanisms is a big question. For continuous integration 
(pony-build) standard output (unittest or nose run with verbose output) 
could be parsed and converted into XML. Not ideal but at least 
demonstrates that it *could* be done without *having* to standardise 
command line handling.

>
> API conformance: The unittest API itself is extraordinarily flexible for
> the sorts of things we're talking about here; I can present a unittest
> TestSuite which will start up instances on EC2 and execute the tests
> there: its very powerful and very lean: define countTestCases(*), run,
> debug(arguably option), and thats nearly all of it: you can, though its
> not desirable, ignore the test result passed into run, and not report
> through the unittest API: your tests will still /run/. Michael Foord,
> others and I all want to make this even better: to make things like
> 'TestRunner' go away (it can now with startTestRun and stopTestRun,
> there is no need for it at all, just hasn't been done in core).
>
>    

Whilst I agree with you (except debug is an abomination that should go 
away) I don't see how it is relevant here. Perhaps just a worthwhile 
observation supporting the load_tests protocol?

> I disagree with Michael on  test_suite being unsupported/whatever: lots
> of people use it. Its nowhere near as convenient as load_tests: its
> kindof a 1st gen attempt at that. Nevertheless, test_suite gets used -
> and supported.
>
>    

Well, it is a legacy protocol. If it goes into distutils2 then it will 
never go away. Test suites that want to be compatible with both can 
simply do:

def load_tests(loader, tests, pattern):
return test_suite()

So I still see no reason to support it in distutils2. Not entirely my 
decision of course.


> Oh, I should note that I don't think setup.py knowing how to do all the
> funky things developers may do is needed: if the primary goals are:
>   - to make it easy for distributors and casual tweak-local-for-me users
>     to be sure the thing builds ok
>   - to make it more routine to incorporate into CI tools like Hudson
>
> then being able to make it do what 'bzr selftest --no-plugins
> --parallel=subprocess -s bt.per_repository -x CommitBuilder --subunit'
> does really isn't important. Being able to (in the first case) make it
> do 'bzr selftest' and (in the second case) 'bzr selftest --subunit |
> subunit2junitxml -o tests.xml' are the important features.
>
> I'm still not clear on the step from 'setup.py test' to 'I can has XML'.
>    

Agreed, not an easy question and still one that is open. Particularly 
relevant for Titus I guess.

Michael

> Hope this helps, I'm happy to put code up if that would help more.
>
> -Rob
>    
>
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20100227/03d786fd/attachment.htm>