[TIP] [tox] Using/building in environments with no internet access

[holger krekel]

Barry,

thanks for your efforts! more below ...

On Wed, Nov 28, 2012 at 16:40 -0500, Barry Warsaw wrote:
> The good news is that we now have python-tox in Debian unstable.  The bad news
> is that the package is actually broken in such a way that it will not
> auto-import into Ubuntu[1].
> 
> I think I've hacked around this, but it's kind of ugly.  This is actually a
> deeper problem too, so I'm looking both for suggestions to improve or
> eliminate my hacks, and for recommendations or policies we can make to ensure
> that packages which run their tests using tox at build time will be able to
> succeed.
> 
> The problem is that we cannot allow tox to download anything from the
> internet, because the buildds have no outside access, including to
> pypi.python.org and pypi.testrun.org.
> 
> I've hacked around this by modifying tox's own tox.ini file to include:
> 
> -----snip snip-----
> [tox]
> envlist=py27,py26,py25,py31,py32,docs
> sitepackages=True
> indexserver =
>     testrun = http://missing.example.com
>     pypi = http://missing.example.com
>     default = http://missing.example.com
> 
> [testenv:X]
> commands=echo {posargs}
> 
> [testenv]
> commands=py.test --junitxml={envlogdir}/junit-{envname}.xml {posargs}
> deps=
> 
> ...
> -----snip snip-----
> 
> TL;DR: I point the various indexservers to non-existent hosts, and null-out
> the [testenv] dependencies.  This *should* be okay because I also set
> sitepackages=True and ensure that the Debian packages providing pytest and py
> are installed into the build chroot[2].
> 
> Now, I want to run tox's own test during the tox package build, but because
> tox isn't installed yet (and because of other Debian-specific issues), I have
> to play some tricks with the make rule:
> 
> test-python%:
> 	HOME=$(FAKEHOME) \
> 	PYTHONPATH=.:/usr/lib/python$*/dist-packages \
> 	python$* -c 'from tox._cmdline import main; main(["-e", "py"+"$*".replace(".", "")])'
> 
> Why set $PYTHONPATH?  Since we're not allowing tox to download from PyPI, but
> we *are* assuring that its dependencies are available (in the Debian-specific
> dist-packages equivalent to site-packages), we have to make sure that when tox
> tries to import packages like py and py.test, it will find the system
> installed versions.  The '.' path element is so that tox can `import tox`.
> 
> Why set $HOME to $FAKEHOME (which is set to `mktemp -dq`)?  It seems that tox
> requires $HOME to exist, but in the buildd environment it gets deliberately
> pointed to a non-existent directory.  Sigh.
> 
> All of this is rather ugly[3], and I shudder to think about making these
> recommendations to other Debian maintainers who want to enable tox-based tests
> in their package builds.  They won't have to play the import game in their
> test-python% rules above, they can just invoke `tox`, though they'll still
> have to -e to only the supported/available versions of Python.  That's fine I
> think.  Forcing the setting of $HOME and $PYTHONPATH and hacking the tox.ini
> to disable dependencies are less okay.

If there is anything specific where i can help, let me know.
I can think of this:

- change tests using $HOME (not many i think) to monkeypatch it to some tempdir
- allow tox to operate with an non-existing $HOME
- introduce a "--ignoredeps" or better a TOX_IGNORE_ALL_DEPS env 
  variable so that tox.ini files don't need to be hacked

Do you think this would help or do you have other suggestions?

The PYTHONPATH issue i don't understand. Isn't that an issue
for other python packages as well?

> If you're still reading, I'd love to get some thoughts on how to make things
> better for build/test environments that have no internet access.  I'm happy to
> file bugs if needed.

I could try to do a 1.4.3 release once we have figured out changes
that would help.

best,
holger

> Cheers,
> -Barry
> 
> [1] (search for "could not install deps")
> https://launchpadlibrarian.net/124241279/buildlog_ubuntu-raring-i386.tox_1.4.2-1_FAILEDTOBUILD.txt.gz
> 
> [2] I also use techniques to prevent local builds from accessing the internet,
> and do test builds in an Ubuntu PPA for more assurances.
> 
> [3] tox 1.4.2-2 should be sponsored into unstable RSN with the above described
> fixes.



> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python