[TIP] Tasting the Cheeseshop

[FELD Boris]

Hi everyone,

I'm one of the students who works on PYTI this summer.

First, let me introduce PYTI. The python testing infrastructure is a Continuous Integration tool able to test python distributions available at PyPI (http://pypi.python.org/) against a set of tests, providing a set of metrics. The project is actually split into different sub-projects and repository:
* PYTI [1]: The master of the orchestra, which will receive PYPI notification of package upload, will download the packages (and its dependencies) and start a worker.
* Goatlog [2]: The report producer, which produce a json execution report (see below why we don't use xmlunit or subunit format).
* Goatlib [3]: The task-manager, which install packages and run them tests.

The PYTI project use VMS for running packages tests as everyone can write harmful python code (like os.command('rm -Rf /*')).

If you want to test it locally, please follow these steps:
* Download & install goatlog (hg clone https://bitbucket.org/marmoute/goatlog; cd goatlog; python setup.py install)

* Download & install goatlib (hg clone https://bitbucket.org/marmoute/goatlib; cd goatlib; python setup.py install)
* Run caprine on an example (cd goatlib/examples/goat; caprine -s basic-bundled.json). You will see logging on stdout and a execution report will be created (at goatlib/examples/goat/log.json).

The other student made an introduction to PYTI at PyconIN, you can watch it here[4].

I was offline for two weeks, so I missed the main conversation, but I'll try to sum-up my thought about it:

As said before, the API using setup.py is quite bad as distutils2 will become the standard. I would use something more simple like:
In order to run test, we must be able to run this piece of python:
> > > import tests
> > > tests.run() #Possibly with arguments
It's simple to run them, and it's easy to make your test library compatible. We can then agree on the list of arguments to use (like produce a xmlunit file, produce a coverage report).
About the main objective of testing PYPI packages, the scope is broader than just running the tests. What we want is to be sure than a user can install it correctly and use it correctly (without bugs and safely). Either junitxml or subunit format is not sufficient. This is why we create Goatlog (documentation[2]), it's a tool we use for creating execution reports (like this one[5]). It includes more information than just the result of the test and we should be able to produce junitxml or subunit from this execution report. It's also why I think we should run the tests after the installation, the fact that tests run before installation adds nothing to the end user.

Integration with Pypi is a very great idea, but in a first time we choose to host execution reports in a different place[6].

The PYTI project continues at a reduced speed, each member of the project is more or less busy.

Regards,
-- 
FELD Boris

[1] http://pyti.readthedocs.org/en/latest/index.html
[2] http://goatlog.readthedocs.org/en/latest/index.html
[3] http://goatlib.readthedocs.org/en/latest/index.html
[4] http://urtalk.kpoint.in/kapsule/gcc-b54c0556-bb9c-465d-8dd9-6f7c80ef5521
[5] http://paste.pocoo.org/show/503388/
[6] http://master.pyti.org/worker