[TIP] RFC: Barely minimal CI + tests + packaging in Python WAS: Guidelines for where to put tests & how to package them

[Olemis Lang]

On Thu, Feb 25, 2010 at 12:01 AM, C. Titus Brown <ctb at msu.edu> wrote:
> Hi all,
>
> here at PyCon there have been a lot of packaging discussions, so I thought
> I'd spend a bit of time outlining some suggestions for where to put
> tests and how to run them.  It's been a bit of a thorn in the side of
> (among other things) continuous integration systems that there's no
> standard way to run Python tests... so let's fix that!
>
[...]
>
> Thoughts?

/me thinking ... Voilà !

Considering my very short experience in the CI field @ testrun.org
(and a few other experiments) I've found that the barely minimal reqs
I'd need to setup and specify a CI build job cleanly (for running
tests ;o) are something like :

  - Prepare the build environment (Solution virtualenv)
  - Use a test runner to output the results in a format understood
    by CI tool. (Solution unittest-xml-reporting read below for the why)
    Further useful reqs are:
    * Use easy_installable package in order to add build step if
      test runner is not provided OOTB by the CI environment
    * It's possible to use a sophisticated framew but I only use
      unittest + doctest, so I prefer to have a small module
      that implements exactly what I need ;o)
    * Easy-to-use, especially not require to hack unittest too much
  - A distutils command to run tests and output the test report
    using the appropriate format (found nothing, but working on this one ;o)
    Further useful reqs are:
    * Selection of the relevant and exact set of tests that should be run
    * Configs for output report
    * Try to install dependencies if some are missing at the CI buildenv
      (read below).
  - A single line invocation of the aforementioned  distutils command

The «Try to install dependencies if some are missing at the CI
buildenv » req is because AFAICR sdist and build don't work if
dependencies are not available (CMIIW) so, if I using a clean isolated
buildenv then the dependencies might not be installed in there. There
are two ways to work around this :

  - Install it globally and inherit global site-packages (IMHO not
    suitable for setups like testrun.org and probably implies
    unnecessary maintenance effort ;o)
  - Include separate build steps to install deps (something that
    seems redundant to me since owner of easy_installable pkg has
    declared such deps in `setup.py` script. Why should he do it
    once again ?)

That's why I think that installing deps before running test (while
executing CI distutils command) might be very useful

Perhaps I'm missing something obvious, so I'll be looking for your feedback

PS: BTW, how could I trigger easy_install(ation) at a given point
while implementing a distutils command, and let the command perform
further actions if deps are installed correctly ?

;o)

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
setUpClass and setUpModule in unittest | Python | Dev  -
http://feedproxy.google.com/~r/TracGViz-full/~3/x18-60vceqg/806136