[TIP] Questions about CI integration testing and packaging with tox and devpi

Oliver Bestwalter oliver at bestwalter.de
Wed Nov 4 14:17:10 PST 2015


Hi,

we are switching from SCM-checkout based distribution to using
packages and I am struggling a bit with how to adjust our testing with
our CI, tox and devpi in the "correct" way. To make my problems clear,
I have to explain what we do and how our workflow is first, so please
bear with me.

I am working in a team that provides a mainly internal (Python) SaaS
to the employees of my company. We are working in a way that we have
short lived feature/bugfix branches (using git-scm) and our master
branches are always deployable (well, let's say 99% of the time ;)).
We deploy (to quite a few different systems) several times a week - if
necessary several times a day. We have our CI set up in a way that we
can just decide that a green build is good to go and deploy directly
from there. This works well, but at the moment we basically just check
out the correct version of the code on the targets and install it via
"pip install -e </path/to/code>". Now we want to make the next step
and use proper packages that can be installed from an internal devpi
server (or later directly from PyPi, when we managed to open-source
parts of our system :)), but keep that basic workflow.

Our testing at the moment is set up in a way that we have several test
stages that are run independently (static, unit, functional and system
tests), Those stages are tox environments (at the moment still with
global setting skipsdist=True), where each tox environment installs
the code repository under test via "pip install -e {toxinidir}" and
then runs the test suite.

But now we want to test the package - not just the code. Otherwise we
would not be really testing what we would deploy. We don't want to
build the package over and over again either, we want to build it
once, when it is first needed and fetch it from devpi on subsequent
runs. There are definitely a gazillion ways to do this, e.g. by
passing build artefacts around on the CI level or by even ignoring the
redundant builds part and just build the package over and over again,
but that feels all wrong and I think it should be possible to handle
this with just tox+devpi. We're not using Jenkins, but I guess, we
could trick our way into using this approach:
http://tox.readthedocs.org/en/latest/example/jenkins.html#access-package-artifacts-between-jenkins-jobs
- but I would prefer not having to do that.

So this would be my dream workflow and ideally it would be handled
mainly automagically by tox+devpi (points 1 and 2 are already taken
care of by our current workflow and CI (Atlassian Bamboo)). I have not
dived in too deep into the building packages and tox/devpi thing yet,
so if this is all wrong and there's a better way: please educate me.

1. Developer creates new branch shiny-new-feature-x and starts pushing code
2. CI automatically picks up shiny-new-feature-x branch and runs the
tests on that branch (basically checking out the branch and invoking
tox -e <flake8|unit|etc.> commands there)
3.
    * The invoked tox process determines the used branch and checks if an
        index for that branch on devpi exists already. If not: the
index is created.
    * Tox retrieves the name of the package under test and generates
the package
        version (from a semantic part provided in the repo by humans
        and the build number passed in by CI via environment variable).
    * If package at version exists on devpi index shiny-new-feature-x already
        (because another tox run (probably on another build machine) built and
        uploaded it already) it installs it from devpi and runs the tests
    * if not it builds it, uploads it to the branch index and then
runs the tests

If this way of working is viable (and worth the fuzz) at all, I would
tend to add a little bit of special casing for the master branch. I
would only want green builds to end up in the prod index, so this
would have to be handled slightly differently. For that purpose it
would be nice to be able to promote a package from one index to a
different one (having a master index, analogous to the branch indexes,
where the packages for the master branch are stored and promoting the
green builds to prod at the end of the testing process).

I would be very grateful to learn to what extent the existing
tox+devpi implementation/interoperation could facilitate this
approach.

cheers
Oliver



More information about the testing-in-python mailing list