[TIP] Tasting the Cheeseshop
exarkun at twistedmatrix.com
exarkun at twistedmatrix.com
Sun Nov 6 20:03:00 PST 2011
On 6 Nov, 03:11 pm, lothiraldan at gmail.com wrote:
>I'm one of the students who works on PYTI this summer.
>First, let me introduce PYTI. The python testing infrastructure is a
>Continuous Integration tool able to test python distributions available
>at PyPI (http://pypi.python.org/) against a set of tests, providing a
>set of metrics. The project is actually split into different sub-
>projects and repository:
>* PYTI : The master of the orchestra, which will receive PYPI
>notification of package upload, will download the packages (and its
>dependencies) and start a worker.
>* Goatlog : The report producer, which produce a json execution
>report (see below why we don't use xmlunit or subunit format).
>* Goatlib : The task-manager, which install packages and run them
>The PYTI project use VMS for running packages tests as everyone can
>write harmful python code (like os.command('rm -Rf /*')).
>If you want to test it locally, please follow these steps:
>* Download & install goatlog (hg clone
>https://bitbucket.org/marmoute/goatlog; cd goatlog; python setup.py
>* Download & install goatlib (hg clone
>https://bitbucket.org/marmoute/goatlib; cd goatlib; python setup.py
>* Run caprine on an example (cd goatlib/examples/goat; caprine -s
>basic-bundled.json). You will see logging on stdout and a execution
>report will be created (at goatlib/examples/goat/log.json).
>The other student made an introduction to PYTI at PyconIN, you can
>watch it here.
>I was offline for two weeks, so I missed the main conversation, but
>I'll try to sum-up my thought about it:
>As said before, the API using setup.py is quite bad as distutils2 will
>become the standard. I would use something more simple like:
>In order to run test, we must be able to run this piece of python:
>> > > import tests
>> > > tests.run() #Possibly with arguments
One problem with this is that it forces every project's tests to collide
in the top-level package namespace.
>It's simple to run them, and it's easy to make your test library
>compatible. We can then agree on the list of arguments to use (like
>produce a xmlunit file, produce a coverage report).
>About the main objective of testing PYPI packages, the scope is broader
>than just running the tests. What we want is to be sure than a user can
>install it correctly and use it correctly (without bugs and safely).
>Either junitxml or subunit format is not sufficient. This is why we
>create Goatlog (documentation), it's a tool we use for creating
>execution reports (like this one). It includes more information than
>just the result of the test and we should be able to produce junitxml
>or subunit from this execution report. It's also why I think we should
>run the tests after the installation, the fact that tests run before
>installation adds nothing to the end user.
Here you run into a problem introduced by the collision I mentioned
above. As soon as you have even two packages installed on a system, how
will you run their tests?
>Integration with Pypi is a very great idea, but in a first time we
>choose to host execution reports in a different place.
>The PYTI project continues at a reduced speed, each member of the
>project is more or less busy.
More information about the testing-in-python