[TIP] Tasting the Cheeseshop

[exarkun at twistedmatrix.com]

On 6 Nov, 03:11 pm, lothiraldan at gmail.com wrote:
>Hi everyone,
>
>I'm one of the students who works on PYTI this summer.
>
>First, let me introduce PYTI. The python testing infrastructure is a 
>Continuous Integration tool able to test python distributions available 
>at PyPI (http://pypi.python.org/) against a set of tests, providing a 
>set of metrics. The project is actually split into different sub- 
>projects and repository:
>* PYTI [1]: The master of the orchestra, which will receive PYPI 
>notification of package upload, will download the packages (and its 
>dependencies) and start a worker.
>* Goatlog [2]: The report producer, which produce a json execution 
>report (see below why we don't use xmlunit or subunit format).
>* Goatlib [3]: The task-manager, which install packages and run them 
>tests.
>
>The PYTI project use VMS for running packages tests as everyone can 
>write harmful python code (like os.command('rm -Rf /*')).
>
>If you want to test it locally, please follow these steps:
>* Download & install goatlog (hg clone 
>https://bitbucket.org/marmoute/goatlog; cd goatlog; python setup.py 
>install)
>
>* Download & install goatlib (hg clone 
>https://bitbucket.org/marmoute/goatlib; cd goatlib; python setup.py 
>install)
>* Run caprine on an example (cd goatlib/examples/goat; caprine -s 
>basic-bundled.json). You will see logging on stdout and a execution 
>report will be created (at goatlib/examples/goat/log.json).
>
>The other student made an introduction to PYTI at PyconIN, you can 
>watch it here[4].
>
>I was offline for two weeks, so I missed the main conversation, but 
>I'll try to sum-up my thought about it:
>
>As said before, the API using setup.py is quite bad as distutils2 will 
>become the standard. I would use something more simple like:
>In order to run test, we must be able to run this piece of python:
>> > > import tests
>> > > tests.run() #Possibly with arguments

One problem with this is that it forces every project's tests to collide 
in the top-level package namespace.
>It's simple to run them, and it's easy to make your test library 
>compatible. We can then agree on the list of arguments to use (like 
>produce a xmlunit file, produce a coverage report).
>About the main objective of testing PYPI packages, the scope is broader 
>than just running the tests. What we want is to be sure than a user can 
>install it correctly and use it correctly (without bugs and safely). 
>Either junitxml or subunit format is not sufficient. This is why we 
>create Goatlog (documentation[2]), it's a tool we use for creating 
>execution reports (like this one[5]). It includes more information than 
>just the result of the test and we should be able to produce junitxml 
>or subunit from this execution report. It's also why I think we should 
>run the tests after the installation, the fact that tests run before 
>installation adds nothing to the end user.

Here you run into a problem introduced by the collision I mentioned 
above.  As soon as you have even two packages installed on a system, how 
will you run their tests?
>Integration with Pypi is a very great idea, but in a first time we 
>choose to host execution reports in a different place[6].
>
>The PYTI project continues at a reduced speed, each member of the 
>project is more or less busy.

Jean-Paul
>Regards,
>--
>FELD Boris
>
>[1] http://pyti.readthedocs.org/en/latest/index.html
>[2] http://goatlog.readthedocs.org/en/latest/index.html
>[3] http://goatlib.readthedocs.org/en/latest/index.html
>[4] http://urtalk.kpoint.in/kapsule/gcc-b54c0556-bb9c-465d- 
>8dd9-6f7c80ef5521
>[5] http://paste.pocoo.org/show/503388/
>[6] http://master.pyti.org/worker