[TIP] Tasting the Cheeseshop
lothiraldan at gmail.com
Sun Nov 6 07:11:54 PST 2011
I'm one of the students who works on PYTI this summer.
First, let me introduce PYTI. The python testing infrastructure is a Continuous Integration tool able to test python distributions available at PyPI (http://pypi.python.org/) against a set of tests, providing a set of metrics. The project is actually split into different sub-projects and repository:
* PYTI : The master of the orchestra, which will receive PYPI notification of package upload, will download the packages (and its dependencies) and start a worker.
* Goatlog : The report producer, which produce a json execution report (see below why we don't use xmlunit or subunit format).
* Goatlib : The task-manager, which install packages and run them tests.
The PYTI project use VMS for running packages tests as everyone can write harmful python code (like os.command('rm -Rf /*')).
If you want to test it locally, please follow these steps:
* Download & install goatlog (hg clone https://bitbucket.org/marmoute/goatlog; cd goatlog; python setup.py install)
* Download & install goatlib (hg clone https://bitbucket.org/marmoute/goatlib; cd goatlib; python setup.py install)
* Run caprine on an example (cd goatlib/examples/goat; caprine -s basic-bundled.json). You will see logging on stdout and a execution report will be created (at goatlib/examples/goat/log.json).
The other student made an introduction to PYTI at PyconIN, you can watch it here.
I was offline for two weeks, so I missed the main conversation, but I'll try to sum-up my thought about it:
As said before, the API using setup.py is quite bad as distutils2 will become the standard. I would use something more simple like:
In order to run test, we must be able to run this piece of python:
> > > import tests
> > > tests.run() #Possibly with arguments
It's simple to run them, and it's easy to make your test library compatible. We can then agree on the list of arguments to use (like produce a xmlunit file, produce a coverage report).
About the main objective of testing PYPI packages, the scope is broader than just running the tests. What we want is to be sure than a user can install it correctly and use it correctly (without bugs and safely). Either junitxml or subunit format is not sufficient. This is why we create Goatlog (documentation), it's a tool we use for creating execution reports (like this one). It includes more information than just the result of the test and we should be able to produce junitxml or subunit from this execution report. It's also why I think we should run the tests after the installation, the fact that tests run before installation adds nothing to the end user.
Integration with Pypi is a very great idea, but in a first time we choose to host execution reports in a different place.
The PYTI project continues at a reduced speed, each member of the project is more or less busy.
More information about the testing-in-python