[TIP] Dependencies installation order with Tox

holger krekel holger at merlinux.eu
Tue Jun 28 11:02:05 PDT 2011 

On Sun, Jun 26, 2011 at 21:35 -0600, Aaron Meurer wrote:
> Well, based on the pip issue you linked to, they won't fix it.  And
> according to the corresponding scipy issue
> (http://projects.scipy.org/scipy/ticket/1429), they won't fix it
> either.  So I don't think it's going to get any love until SciPy moves
> to static metadata (and who knows when that will happen).

ups, bah.

> Is there perhaps some way I could "trick" tox into installing numpy
> first or separately.  Maybe I could take advantage of the caching
> somehow?  Or is there more advanced syntax available for deps (e.g.,
> if I could get it to run an arbitrary command, then I could do this)?

there are some tricks we could do.  Mayber it'd be good to allow 
defining the "install" step and allow it to consist of multiple 
commands so that you could effectively do something custom like:

    pip install [OPTS] numpy
    pip install [OPTS] scipy

Where OPTS would be the currently computed options (from various testenv vars)
However, there are other tox users wanting to specify other certain (pip)
options.

maybe it's easiest we could an option "install_deps_separately = True"
which would result in the above. Are you up for implementing this
solution?  You could take a look at https://bitbucket.org/jorgeecardona/tox
for a similar feature.

> and relying on my non-tox, systemwide testing to test for scipy
> compatibility (fortunately, in SymPy, the library I am testing, scipy
> usage is completely optional and very minimal).

might be your best current option. I am on travel and parental leave
currently and only am hacking from time to time.  Maybe best to spend 
my time reviewing suggestions and contributions.

best,
holger

> Aaron Meurer
> 
> On Sun, Jun 26, 2011 at 4:30 AM, holger krekel <holger at merlinux.eu> wrote:
> > On Mon, Jun 20, 2011 at 14:50 -0600, Aaron Meurer wrote:
> >> Right.  I rather suspected the problem was with SciPy.
> >>
> >> Is there a way around this?  The way it is, I can't test with both
> >> scipy and numpy in tox.
> >
> > I don't know of an immediate work around i am afraid.
> > Maybe you could raise the question with the people on the
> > pip related issue.
> >
> > best,
> > holger
> >
> >> Aaron Meurer
> >>
> >> On Mon, Jun 20, 2011 at 9:39 AM, holger krekel <holger at merlinux.eu> wrote:
> >> > Hi Aaron,
> >> >
> >> > gratefully, Jannis Leidel pointed me to this issue:
> >> >
> >> >    https://github.com/pypa/pip/issues/25
> >> >
> >> > which indicates and summarizes the problems you are seein.
> >> > The conclusion there is that this is a problem of scipy.
> >> >
> >> > best,
> >> > holger
> >> >
> >> > On Mon, Jun 20, 2011 at 01:47 -0600, Aaron Meurer wrote:
> >> >> No, it does not.  I get
> >> >>
> >> >> Downloading/unpacking scipy
> >> >>   Downloading scipy-0.9.0.zip (7.6Mb): 7.6Mb downloaded
> >> >>   Running setup.py egg_info for package scipy
> >> >>     Traceback (most recent call last):
> >> >>       File "<string>", line 14, in <module>
> >> >>       File "/Users/aaronmeurer/Desktop/py25/build/scipy/setup.py",
> >> >> line 181, in <module>
> >> >>         setup_package()
> >> >>       File "/Users/aaronmeurer/Desktop/py25/build/scipy/setup.py",
> >> >> line 131, in setup_package
> >> >>         from numpy.distutils.core import setup
> >> >>     ImportError: No module named numpy.distutils.core
> >> >>     Complete output from command python setup.py egg_info:
> >> >>     Traceback (most recent call last):
> >> >>
> >> >>   File "<string>", line 14, in <module>
> >> >>
> >> >>   File "/Users/aaronmeurer/Desktop/py25/build/scipy/setup.py", line
> >> >> 181, in <module>
> >> >>
> >> >>     setup_package()
> >> >>
> >> >>   File "/Users/aaronmeurer/Desktop/py25/build/scipy/setup.py", line
> >> >> 131, in setup_package
> >> >>
> >> >>     from numpy.distutils.core import setup
> >> >>
> >> >> ImportError: No module named numpy.distutils.core
> >> >>
> >> >> ----------------------------------------
> >> >> Command python setup.py egg_info failed with error code 1
> >> >> Storing complete log in /Users/aaronmeurer/.pip/pip.log
> >> >>
> >> >> which is basically the same error that tox gave me.  But if I do
> >> >>
> >> >> py25/bin/pip install numpy
> >> >> py25/bin/pip install scipy
> >> >>
> >> >> it works (actually, scipy exits with a compiler error, but I think I
> >> >> can fix that by making it compile an older version, and anyway it's
> >> >> unrelated).
> >> >>
> >> >> I know almost nothing about how this stuff works, but if I had to
> >> >> guess, I'd say scipy isn't listing numpy as a dependency correctly.
> >> >> The question is, why does pip insist on installing scipy first, even
> >> >> when it is given second?
> >> >>
> >> >> Aaron Meurer
> >> >>
> >> >> On Mon, Jun 20, 2011 at 1:28 AM, holger krekel <holger at merlinux.eu> wrote:
> >> >> > Hi Aaron,
> >> >> >
> >> >> > if you simply do:
> >> >> >
> >> >> >    virtualenv --distribute --no-site-packages -p /sw/bin/python2.5 py25
> >> >> >    py25/bin/pip install numpy scipy
> >> >> >
> >> >> > does it work?  If so, can you remove .tox (to remove the download cache)
> >> >> > and re-invoke it?
> >> >> >
> >> >> > best,
> >> >> > holger
> >> >> >
> >> >> > On Sun, Jun 19, 2011 at 19:26 -0600, Aaron Meurer wrote:
> >> >> >> Hi again.
> >> >> >>
> >> >> >> I'm having another problem with dependencies with tox.  I'm trying to
> >> >> >> test using the dependencies numpy and scipy, but they do not install
> >> >> >> properly.  From what I can tell, scipy requires numpy, but it attempts
> >> >> >> to install scipy first, and so the whole thing fails.  This happens
> >> >> >> even though I have numpy listed first in the deps list.
> >> >> >>
> >> >> >> I have attached a sample tox.ini file and output of the command tox -c
> >> >> >> tox.ini.test -v -r to demonstrate.  Note the basepython is just to
> >> >> >> ensure it uses fink's Python on my machine, which has numpy and scipy
> >> >> >> installed. I have both Python 2.5 and 2.7 just to see if it would
> >> >> >> work, but it appears that the behavior in both is the same.
> >> >> >>
> >> >> >> Aaron Meurer
> >> >> >
> >> >> >> [TOX] emptying /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/log
> >> >> >> [TOX] using tox.ini: /Users/aaronmeurer/Documents/python/sympy/sympy/tox.ini.test
> >> >> >> [TOX] using tox-0.9 from /Library/Python/2.6/site-packages/tox-0.9-py2.6.egg/tox/__init__.pyc
> >> >> >> ______________________________________________________________________________________________ [tox sdist] _______________________________________________________________________________________________
> >> >> >> [TOX] ***creating sdist package
> >> >> >> [TOX] emptying /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/dist
> >> >> >> [TOX] /Users/aaronmeurer/Documents/python/sympy/sympy$ /usr/bin/python setup.py sdist --formats=zip --dist-dir .tox/dist >.tox/log/0.log
> >> >> >> [TOX] ***copying new sdistfile to '/Users/aaronmeurer/.tox/distshare/sympy-0.7.0.rc2.zip'
> >> >> >> ___________________________________________________________________________________________ [tox testenv:py25] ___________________________________________________________________________________________
> >> >> >> [TOX] ***recreating virtualenv py25 (configchange/incomplete install detected)
> >> >> >> [TOX] emptying /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25
> >> >> >> [TOX] /Users/aaronmeurer/Documents/python/sympy/sympy/.tox$ virtualenv --distribute --no-site-packages -p /sw/bin/python2.5 py25 >py25/log/0.log
> >> >> >> [TOX] ***installing dependencies: numpy, scipy
> >> >> >> [TOX] /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/log$ ../bin/pip install --download-cache=/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download numpy scipy >1.log
> >> >> >> [TOX] ERROR: invocation failed, logfile: /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/log/1.log
> >> >> >> [TOX] ERROR: /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/log$ ../bin/pip install --download-cache=/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download numpy scipy >1.log
> >> >> >> Downloading/unpacking scipy
> >> >> >>   Using download cache from /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download/http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fs%2Fscipy%2Fscipy-0.9.0.zip
> >> >> >>   Running setup.py egg_info for package scipy
> >> >> >>     Traceback (most recent call last):
> >> >> >>       File "<string>", line 14, in <module>
> >> >> >>       File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/build/scipy/setup.py", line 181, in <module>
> >> >> >>         setup_package()
> >> >> >>       File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/build/scipy/setup.py", line 131, in setup_package
> >> >> >>         from numpy.distutils.core import setup
> >> >> >>     ImportError: No module named numpy.distutils.core
> >> >> >>     Complete output from command python setup.py egg_info:
> >> >> >>     Traceback (most recent call last):
> >> >> >>
> >> >> >>   File "<string>", line 14, in <module>
> >> >> >>
> >> >> >>   File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/build/scipy/setup.py", line 181, in <module>
> >> >> >>
> >> >> >>     setup_package()
> >> >> >>
> >> >> >>   File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/build/scipy/setup.py", line 131, in setup_package
> >> >> >>
> >> >> >>     from numpy.distutils.core import setup
> >> >> >>
> >> >> >> ImportError: No module named numpy.distutils.core
> >> >> >>
> >> >> >> ----------------------------------------
> >> >> >> Command python setup.py egg_info failed with error code 1
> >> >> >> Storing complete log in /Users/aaronmeurer/.pip/pip.log
> >> >> >>
> >> >> >> [TOX] ERROR: could not install deps [numpy, scipy]
> >> >> >> ___________________________________________________________________________________________ [tox testenv:py27] ___________________________________________________________________________________________
> >> >> >> [TOX] ***recreating virtualenv py27 (configchange/incomplete install detected)
> >> >> >> [TOX] emptying /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27
> >> >> >> [TOX] /Users/aaronmeurer/Documents/python/sympy/sympy/.tox$ virtualenv --distribute --no-site-packages -p /sw/bin/python2.7 py27 >py27/log/0.log
> >> >> >> [TOX] ***installing dependencies: numpy, scipy
> >> >> >> [TOX] /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/log$ ../bin/pip install --download-cache=/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download numpy scipy >1.log
> >> >> >> [TOX] ERROR: invocation failed, logfile: /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/log/1.log
> >> >> >> [TOX] ERROR: /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/log$ ../bin/pip install --download-cache=/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download numpy scipy >1.log
> >> >> >> Downloading/unpacking scipy
> >> >> >>   Using download cache from /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download/http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fs%2Fscipy%2Fscipy-0.9.0.zip
> >> >> >>   Running setup.py egg_info for package scipy
> >> >> >>     Traceback (most recent call last):
> >> >> >>       File "<string>", line 14, in <module>
> >> >> >>       File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/build/scipy/setup.py", line 181, in <module>
> >> >> >>         setup_package()
> >> >> >>       File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/build/scipy/setup.py", line 131, in setup_package
> >> >> >>         from numpy.distutils.core import setup
> >> >> >>     ImportError: No module named numpy.distutils.core
> >> >> >>     Complete output from command python setup.py egg_info:
> >> >> >>     Traceback (most recent call last):
> >> >> >>
> >> >> >>   File "<string>", line 14, in <module>
> >> >> >>
> >> >> >>   File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/build/scipy/setup.py", line 181, in <module>
> >> >> >>
> >> >> >>     setup_package()
> >> >> >>
> >> >> >>   File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/build/scipy/setup.py", line 131, in setup_package
> >> >> >>
> >> >> >>     from numpy.distutils.core import setup
> >> >> >>
> >> >> >> ImportError: No module named numpy.distutils.core
> >> >> >>
> >> >> >> ----------------------------------------
> >> >> >> Command python setup.py egg_info failed with error code 1
> >> >> >> Storing complete log in /Users/aaronmeurer/.pip/pip.log
> >> >> >>
> >> >> >> [TOX] ERROR: could not install deps [numpy, scipy]
> >> >> >> _____________________________________________________________________________________________ [tox summary] ______________________________________________________________________________________________
> >> >> >> [TOX] ERROR: py25: could not install deps [numpy, scipy]
> >> >> >> [TOX] ERROR: py27: could not install deps [numpy, scipy]
> >> >> >
> >> >> >
> >> >> >> _______________________________________________
> >> >> >> testing-in-python mailing list
> >> >> >> testing-in-python at lists.idyll.org
> >> >> >> http://lists.idyll.org/listinfo/testing-in-python
> >> >> >
> >> >> >
> >> >>
> >> >
> >>
> >
>

More information about the testing-in-python mailing list