[TIP] Defining a convention for running tests, #2

[Robert Collins]

On Sat, 2010-02-27 at 12:13 -0800, C. Titus Brown wrote:
> Hi all,
> 
> I've updated the README for my SomePackage example code,

So, my key points are, I guess:

python -m somepackage.tests.run - isn't very useful:
 - doesn't define getting JUnitXML (is it a command line option? should
you use subunit? should you use nose-junit, should you use
xmltestrunner?)

 - doesn't define command line parameters except implicitly as 'the
unittest ones' based on your sample code; in which case aiming for API
conformance is more useful.

These concerns also apply to setup.py test. As you say '
 - Titus Brown: I also don't really understand what the arguments to
   'setup.py test' are.'
For integration with CI tools, the primary, defacto method is 'get a
JUnit XML file on disk', with everything else secondary.


API conformance: The unittest API itself is extraordinarily flexible for
the sorts of things we're talking about here; I can present a unittest
TestSuite which will start up instances on EC2 and execute the tests
there: its very powerful and very lean: define countTestCases(*), run,
debug(arguably option), and thats nearly all of it: you can, though its
not desirable, ignore the test result passed into run, and not report
through the unittest API: your tests will still /run/. Michael Foord,
others and I all want to make this even better: to make things like
'TestRunner' go away (it can now with startTestRun and stopTestRun,
there is no need for it at all, just hasn't been done in core).


I disagree with Michael on  test_suite being unsupported/whatever: lots
of people use it. Its nowhere near as convenient as load_tests: its
kindof a 1st gen attempt at that. Nevertheless, test_suite gets used -
and supported.


Oh, I should note that I don't think setup.py knowing how to do all the
funky things developers may do is needed: if the primary goals are:
 - to make it easy for distributors and casual tweak-local-for-me users
   to be sure the thing builds ok
 - to make it more routine to incorporate into CI tools like Hudson

then being able to make it do what 'bzr selftest --no-plugins
--parallel=subprocess -s bt.per_repository -x CommitBuilder --subunit'
does really isn't important. Being able to (in the first case) make it
do 'bzr selftest' and (in the second case) 'bzr selftest --subunit |
subunit2junitxml -o tests.xml' are the important features.

I'm still not clear on the step from 'setup.py test' to 'I can has XML'.

Hope this helps, I'm happy to put code up if that would help more.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20100228/33d24320/attachment.pgp>