[TIP] (RFC) multi-dimensional/variant tox configuration (V1)

holger krekel holger at merlinux.eu
Sat Jul 7 06:58:53 PDT 2012

Hi tox users,

I'd like to find a good way to introduce multi-dimensional configuration
to tox.ini files.  I have written up a draft idea on how to do it and
would appreciate feedback.  I provide two examples of transformed tox.ini
files.  If you have suggestions (or example tox.ini which you would like
to transform) i'd be grateful to hear about them.

best & thanks,

P.S.: Ronny Pfannschmidt just posted a different way to handle
variants with tox, i'll check it up together with other feedback.

V1 draft: multi-dimensional configuration with tox


- there is no way to define dependency or other variants in tox.ini files, 
  instead you have explicitely spell out all combinations in 
  separate testenvs. Examples:


- tox always uses pip currently.  So there is no check for your packages
  that installing with easy_install will work. Moreover, some packages,
  like greenlet on Win32, require easy_install if you have no suitable
  C-compiler on the machine.  Tox cannot be used then currently.


- allow to more easily define and run dependency/interpreter variants 
  with testenvs
- allow to run variants of installing via easy_install or pip. 

Generating and selecting variants

Suppose you want to test your package against mypkg-1.3 and mypkg-1.4
framework versions and against python2.6,2.7,pypy-1.9 and 3.2 interpreters. 
Today you would have to create 2*4 = 8 ``[testenv*]`` sections to instruct
tox to test against all of them.  With tox-1.X you can use a new mechanism
which is based on two ideas:

* allow to specify partial testenv values in [testenv-VARIANT] sections
* introduce "generator" expressions to the "envlist" setting to ease
  enumerating all variant combinations.

Without much further introduction, here is an example ``tox.ini``::

    envlist = 
        deps = nose
        commands = nosetests

        +deps = mypkg<1.4

        +deps = mypkg<1.5

If you don't want to run django-mypkg with pypy the envlist would look like

    envlist = 

Generator expressions in the envlist setting

Generator expressions in the ``envlist`` work like this:

* ``[...]`` parts contain a comma-separated list of names. Each name
  will generate a new environment reference. 
* repeat the process until there are no more generator expressions

Variant specification with [testenv-VARNAME]

The ``[testenv-mypkg13]`` and ``[testenv-mypkg14]`` can define
settings for the respective variant.  Their specification
can use a new ``+deps`` setting to allow for appending dependencies rather than replacing it.

Showing all expanded sections

To help with understanding how the variants will produce section values,
you can ask tox to show their expansion with a new option:

    $ tox -l
    [XXX output ommitted for now]

Making sure your packages installs with easy_install

The new "installer" testenv setting allows to specify the tool for installation:

    installer = easy_install

If you want to have your package installed with both easy_install
and pip, you can use variants again:

    installer = easy_install

    installer = pip

    envlist = py[26,27,32]-django[13,14]-[easy,pip]

Note that tox comes with some predefined variants, namely:

    - [easy,pip] use easy_install or pip
    - [py24,py25,py26,py27,py31,py32,pypy,jy] use the respective pythonNN
      or PyPy or Jython interpreter 

You can use those in your "envlist" specification without the need to
define them yourself.
Transforming the examples: django-rest

The original `tox.ini <http://code.larlet.fr/django-rest-framework/src/eed0f39a7e45/tox.ini>`_ file has 159 lines and a lot of repetition, the new one would 
have 26 lines and almost no repetition::

    envlist = py[25,26,27]-django[12,13]-[example]

    commands = python setup.py test


    +deps= django==1.2.4

    +deps= django==1.2.4

    deps = 

    commands = python examples/runtests.py

Apart from the much more concise specification, it is now also easy to add further variants like testing installation with easy_install in addition
to pip.

Transforming the examples: django-treebeard

Another `tox.ini <https://bitbucket.org/tabo/django-treebeard/raw/93b579395a9c/tox.ini>`_ has 233 lines and runs tests against multiple Postgres and Mysql engines in the same testenv section.  We can easily split this and reduce to 39 non-repetive lines::

    envlist =

    changedir = docs
    deps =
    commands =
      make clean
      make html


    commands =
      {envpython} runtests.py {posargs}

    +deps = psycopg2
    commands =
      {envpython} runtests.py --DATABASE_ENGINE=postgresql_psycopg2 --DATABASE_USER=postgres {posargs}

    +deps = MySQL-python
    commands =
      {envpython} runtests.py --DATABASE_ENGINE=mysql --DATABASE_USER=root {posargs}

    +deps = Django==1.0.4

More information about the testing-in-python mailing list