[TIP] [tox] Using/building in environments with no internet access

[holger krekel]

On Wed, Nov 28, 2012 at 18:00 -0500, Barry Warsaw wrote:
> On Nov 28, 2012, at 10:09 PM, holger krekel wrote:
> 
> >thanks for your efforts! more below ...
> 
> I can't tell you how many people have sent me +1s for working on making tox
> available in the Debian ecosystem.  It's for purely selfish reasons, I assure
> you. :)
> 
> >If there is anything specific where i can help, let me know.  I can think of
> >this:
> >
> >- change tests using $HOME (not many i think) to monkeypatch it to some
> >  tempdir
> >- allow tox to operate with an non-existing $HOME
> 
> Solving the $HOME requirement is a minor issue, but it's an annoyance that
> would be nice to fix.  Anything you can do here to avoid having to point $HOME
> at a tempdir during the build, and tox's own tests would be great.

Actually I just tried to reproduce this issue - i set HOME to
/tmp/qweoqwie and ran tox tests both on trunk and with 1.4.2. Worked
without failure.  I must be missing some detail.

> >- introduce a "--ignoredeps" or better a TOX_IGNORE_ALL_DEPS env 
> >  variable so that tox.ini files don't need to be hacked
> >
> >Do you think this would help or do you have other suggestions?
> 
> That would definitely help, but I'm not sure it quite goes all the way.  We do
> want tox to ignore tox.ini dependencies (and either or both of the switch and
> envar would be fine, with a mild preference on the envar if it has to be one
> or the other).
> 
> However, we also need tox to use the system Python's packages in those cases.
> Maybe you're thinking about that too.  For tox itself, we have to ignore the
> py and py.test deps, but also have to avoid downloading the tox .zip file
> which I think it does so that it can `import tox`.  More on that below.
> 
> So if --ignoredeps/$TOX_IGNORE_ALL_DEPS just lets tox trust Python's sys.path,
> I think that would work.  But note the use of dist-packages below.

TOX_TRUST_GLOBAL_SITEPACKAGES=1 could lead tox to force "sitepackages=True"
in all envs and don't install any deps despite tox.ini settings.

> >The PYTHONPATH issue i don't understand. Isn't that an issue
> >for other python packages as well?
> 
> It's not, although we currently have no packages with build-time tests
> depending on tox, because until now, we didn't have tox available.  Probably,
> most packages-with-build-time-tests use `python setup.py test` or nosetests,
> and because these use the system Python's sys.path, they all succeed just
> fine, assuming of course, we've done our job to ensure the proper build
> dependencies are installed.
> 
> For some reason, this doesn't work with tox's tests.
> 
> Without adding '.' the `import tox` fails.  This probably makes the most sense
> because tox itself isn't installed at the time its tests are run.  So this
> allows us to import from the unpacked zip.  It's cheating a bit, but seems
> okay, and would only be required for tox's build, since it's a bootstrapping
> problem.  Other packages would just build-depend on tox and then they should
> be able to run the equivalent of:
> 
>     for short_py_version in all_available_versions:
>         # E.g. 2.7 -> py27
>         run('tox -e py{}'.format(short_py_version.replace('.', '')))
> 
> Without adding /usr/lib/pythonX.Y/dist-packages, you get lots of failures like
> this when running tox's own tests:
> 
> ____________________________ test_separate_sdist ______________________________
> 
> cmd = <conftest.Cmd instance at 0x1e2a7a0>
> initproj = <function initproj at 0x1e23758>
> 
>     def test_separate_sdist(cmd, initproj):
>         distshare = cmd.tmpdir.join("distshare")
>         initproj("pkg123-0.7", filedefs={
>             'tox.ini': """
>                 [tox]
>                 distshare=%s
>                 sdistsrc={distshare}/pkg123-0.7.zip
>             """ % distshare
>         })
>         result = cmd.run("tox", "--sdistonly")
> >       assert not result.ret
> E       assert not 1
> E        +  where 1 = <conftest.RunResult instance at 0x1e49248>.ret
> 
> /build/tox-DkDcdQ/tox-1.4.2/tests/test_z_cmdline.py:417: AssertionError
> ------------------------------- Captured stdout --------------------------------
> created project in /tmp/pytest-18/test_separate_sdist0/pkg123
> /tmp/pytest-18/test_separate_sdist0/pkg123$ tox --sdistonly
> ------------------------------- Captured stderr --------------------------------
> Traceback (most recent call last):
>   File "/build/tox-DkDcdQ/tox-1.4.2/.tox/py27/bin/tox", line 5, in <module>
>     from pkg_resources import load_entry_point
>   File "/build/tox-DkDcdQ/tox-1.4.2/.tox/py27/local/lib/python2.7/site-packages/distribute-0.6.30-py2.7.egg/pkg_resources.py", line 2807, in <module>
>     working_set.require(__requires__)
>   File "/build/tox-DkDcdQ/tox-1.4.2/.tox/py27/local/lib/python2.7/site-packages/distribute-0.6.30-py2.7.egg/pkg_resources.py", line 690, in require
>     needed = self.resolve(parse_requirements(requirements))
>   File "/build/tox-DkDcdQ/tox-1.4.2/.tox/py27/local/lib/python2.7/site-packages/distribute-0.6.30-py2.7.egg/pkg_resources.py", line 588, in resolve
>     raise DistributionNotFound(req)
> pkg_resources.DistributionNotFound: py>=1.4.9
> 
> 
> 'site-packages' here is a clue because on Debian, we install system packages
> into dist-packages, and have our own hacked version of distribute to search
> dist-packages in addition to site-packages (for both good and crazy reasons I
> won't get into here ;).
> 
> So my guess is that maybe tox is installing its own version of distribute that
> doesn't know about the Debian dist-packages thing?  Or, the tox in the
> virtualenv's bin doesn't use the same sys.path as system Python.

Tox doesn't directly install distribute.  It invokes "virtualenv" which
does that.  It also doesn't install virtualenv, it's one of the installation
dependencies.  I am not sure why the debian-distribute version is not ysed.

Tox own tests depend on the "tox" command being available and findable
on PATH.  In the above tracebsack it finds it is found in
"/build/tox-DkDcdQ/tox-1.4.2/.tox/py27/bin/tox" - but i wonder why.
In order for that ".tox/py27" to exist, tox must have been invoked already.

In any case, tox own tests are written based on the assumption that one did a
"python setup.py develop" so that the "tox" command is available and
can be invoked.  That is an assumption that is simply not valid
in the debian build env for tox itself, right?

Other packages using tox for testing should indeed not have that problem.

> >I could try to do a 1.4.3 release once we have figured out changes
> >that would help.
> 
> One other thing that would be helpful actually, is if you could upload a
> tar.gz to PyPI in addition to the .zip file.  We have to repack zips.  It's
> not a showstopper, but a minor annoyance.

Will go for that for the next release.  There is no issue using tgz on
windows so i best just keep it at a single tgz package format.

best,
holger