[TIP] Dependencies installation order with Tox

Aaron Meurer asmeurer at gmail.com
Tue Jun 28 17:19:10 PDT 2011


On Tue, Jun 28, 2011 at 12:02 PM, holger krekel <holger at merlinux.eu> wrote:
> On Sun, Jun 26, 2011 at 21:35 -0600, Aaron Meurer wrote:
>> Well, based on the pip issue you linked to, they won't fix it.  And
>> according to the corresponding scipy issue
>> (http://projects.scipy.org/scipy/ticket/1429), they won't fix it
>> either.  So I don't think it's going to get any love until SciPy moves
>> to static metadata (and who knows when that will happen).
>
> ups, bah.
>
>> Is there perhaps some way I could "trick" tox into installing numpy
>> first or separately.  Maybe I could take advantage of the caching
>> somehow?  Or is there more advanced syntax available for deps (e.g.,
>> if I could get it to run an arbitrary command, then I could do this)?
>
> there are some tricks we could do.  Mayber it'd be good to allow
> defining the "install" step and allow it to consist of multiple
> commands so that you could effectively do something custom like:
>
>    pip install [OPTS] numpy
>    pip install [OPTS] scipy
>
> Where OPTS would be the currently computed options (from various testenv vars)
> However, there are other tox users wanting to specify other certain (pip)
> options.
>
> maybe it's easiest we could an option "install_deps_separately = True"
> which would result in the above. Are you up for implementing this
> solution?  You could take a look at https://bitbucket.org/jorgeecardona/tox
> for a similar feature.

I can attempt it, though no promises. The code doesn't seem to be as
well documented as I would like (OK, I'll be honest, it isn't
documented at all). Perhaps you could jumpstart me by pointing to the
file where it would go?

Although probably the biggest hurdle for me will be learning how to
use hg.  I know git pretty well, but that's it as far as my VCS
fluency.  Maybe I can figure it out using
https://github.com/sympy/sympy/wiki/git-hg-rosetta-stone.  At any
rate, I could just send you an old fashioned patch file if I can't
figure it out.

Before I do anything, which method should I aim to implement?  Should
I go for "install_deps_separately", or should I try to do something
more advanced?

Aaron Meurer

>
>> and relying on my non-tox, systemwide testing to test for scipy
>> compatibility (fortunately, in SymPy, the library I am testing, scipy
>> usage is completely optional and very minimal).
>
> might be your best current option. I am on travel and parental leave
> currently and only am hacking from time to time.  Maybe best to spend
> my time reviewing suggestions and contributions.
>
> best,
> holger
>
>> Aaron Meurer
>>
>> On Sun, Jun 26, 2011 at 4:30 AM, holger krekel <holger at merlinux.eu> wrote:
>> > On Mon, Jun 20, 2011 at 14:50 -0600, Aaron Meurer wrote:
>> >> Right.  I rather suspected the problem was with SciPy.
>> >>
>> >> Is there a way around this?  The way it is, I can't test with both
>> >> scipy and numpy in tox.
>> >
>> > I don't know of an immediate work around i am afraid.
>> > Maybe you could raise the question with the people on the
>> > pip related issue.
>> >
>> > best,
>> > holger
>> >
>> >> Aaron Meurer
>> >>
>> >> On Mon, Jun 20, 2011 at 9:39 AM, holger krekel <holger at merlinux.eu> wrote:
>> >> > Hi Aaron,
>> >> >
>> >> > gratefully, Jannis Leidel pointed me to this issue:
>> >> >
>> >> >    https://github.com/pypa/pip/issues/25
>> >> >
>> >> > which indicates and summarizes the problems you are seein.
>> >> > The conclusion there is that this is a problem of scipy.
>> >> >
>> >> > best,
>> >> > holger
>> >> >
>> >> > On Mon, Jun 20, 2011 at 01:47 -0600, Aaron Meurer wrote:
>> >> >> No, it does not.  I get
>> >> >>
>> >> >> Downloading/unpacking scipy
>> >> >>   Downloading scipy-0.9.0.zip (7.6Mb): 7.6Mb downloaded
>> >> >>   Running setup.py egg_info for package scipy
>> >> >>     Traceback (most recent call last):
>> >> >>       File "<string>", line 14, in <module>
>> >> >>       File "/Users/aaronmeurer/Desktop/py25/build/scipy/setup.py",
>> >> >> line 181, in <module>
>> >> >>         setup_package()
>> >> >>       File "/Users/aaronmeurer/Desktop/py25/build/scipy/setup.py",
>> >> >> line 131, in setup_package
>> >> >>         from numpy.distutils.core import setup
>> >> >>     ImportError: No module named numpy.distutils.core
>> >> >>     Complete output from command python setup.py egg_info:
>> >> >>     Traceback (most recent call last):
>> >> >>
>> >> >>   File "<string>", line 14, in <module>
>> >> >>
>> >> >>   File "/Users/aaronmeurer/Desktop/py25/build/scipy/setup.py", line
>> >> >> 181, in <module>
>> >> >>
>> >> >>     setup_package()
>> >> >>
>> >> >>   File "/Users/aaronmeurer/Desktop/py25/build/scipy/setup.py", line
>> >> >> 131, in setup_package
>> >> >>
>> >> >>     from numpy.distutils.core import setup
>> >> >>
>> >> >> ImportError: No module named numpy.distutils.core
>> >> >>
>> >> >> ----------------------------------------
>> >> >> Command python setup.py egg_info failed with error code 1
>> >> >> Storing complete log in /Users/aaronmeurer/.pip/pip.log
>> >> >>
>> >> >> which is basically the same error that tox gave me.  But if I do
>> >> >>
>> >> >> py25/bin/pip install numpy
>> >> >> py25/bin/pip install scipy
>> >> >>
>> >> >> it works (actually, scipy exits with a compiler error, but I think I
>> >> >> can fix that by making it compile an older version, and anyway it's
>> >> >> unrelated).
>> >> >>
>> >> >> I know almost nothing about how this stuff works, but if I had to
>> >> >> guess, I'd say scipy isn't listing numpy as a dependency correctly.
>> >> >> The question is, why does pip insist on installing scipy first, even
>> >> >> when it is given second?
>> >> >>
>> >> >> Aaron Meurer
>> >> >>
>> >> >> On Mon, Jun 20, 2011 at 1:28 AM, holger krekel <holger at merlinux.eu> wrote:
>> >> >> > Hi Aaron,
>> >> >> >
>> >> >> > if you simply do:
>> >> >> >
>> >> >> >    virtualenv --distribute --no-site-packages -p /sw/bin/python2.5 py25
>> >> >> >    py25/bin/pip install numpy scipy
>> >> >> >
>> >> >> > does it work?  If so, can you remove .tox (to remove the download cache)
>> >> >> > and re-invoke it?
>> >> >> >
>> >> >> > best,
>> >> >> > holger
>> >> >> >
>> >> >> > On Sun, Jun 19, 2011 at 19:26 -0600, Aaron Meurer wrote:
>> >> >> >> Hi again.
>> >> >> >>
>> >> >> >> I'm having another problem with dependencies with tox.  I'm trying to
>> >> >> >> test using the dependencies numpy and scipy, but they do not install
>> >> >> >> properly.  From what I can tell, scipy requires numpy, but it attempts
>> >> >> >> to install scipy first, and so the whole thing fails.  This happens
>> >> >> >> even though I have numpy listed first in the deps list.
>> >> >> >>
>> >> >> >> I have attached a sample tox.ini file and output of the command tox -c
>> >> >> >> tox.ini.test -v -r to demonstrate.  Note the basepython is just to
>> >> >> >> ensure it uses fink's Python on my machine, which has numpy and scipy
>> >> >> >> installed. I have both Python 2.5 and 2.7 just to see if it would
>> >> >> >> work, but it appears that the behavior in both is the same.
>> >> >> >>
>> >> >> >> Aaron Meurer
>> >> >> >
>> >> >> >> [TOX] emptying /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/log
>> >> >> >> [TOX] using tox.ini: /Users/aaronmeurer/Documents/python/sympy/sympy/tox.ini.test
>> >> >> >> [TOX] using tox-0.9 from /Library/Python/2.6/site-packages/tox-0.9-py2.6.egg/tox/__init__.pyc
>> >> >> >> ______________________________________________________________________________________________ [tox sdist] _______________________________________________________________________________________________
>> >> >> >> [TOX] ***creating sdist package
>> >> >> >> [TOX] emptying /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/dist
>> >> >> >> [TOX] /Users/aaronmeurer/Documents/python/sympy/sympy$ /usr/bin/python setup.py sdist --formats=zip --dist-dir .tox/dist >.tox/log/0.log
>> >> >> >> [TOX] ***copying new sdistfile to '/Users/aaronmeurer/.tox/distshare/sympy-0.7.0.rc2.zip'
>> >> >> >> ___________________________________________________________________________________________ [tox testenv:py25] ___________________________________________________________________________________________
>> >> >> >> [TOX] ***recreating virtualenv py25 (configchange/incomplete install detected)
>> >> >> >> [TOX] emptying /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25
>> >> >> >> [TOX] /Users/aaronmeurer/Documents/python/sympy/sympy/.tox$ virtualenv --distribute --no-site-packages -p /sw/bin/python2.5 py25 >py25/log/0.log
>> >> >> >> [TOX] ***installing dependencies: numpy, scipy
>> >> >> >> [TOX] /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/log$ ../bin/pip install --download-cache=/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download numpy scipy >1.log
>> >> >> >> [TOX] ERROR: invocation failed, logfile: /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/log/1.log
>> >> >> >> [TOX] ERROR: /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/log$ ../bin/pip install --download-cache=/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download numpy scipy >1.log
>> >> >> >> Downloading/unpacking scipy
>> >> >> >>   Using download cache from /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download/http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fs%2Fscipy%2Fscipy-0.9.0.zip
>> >> >> >>   Running setup.py egg_info for package scipy
>> >> >> >>     Traceback (most recent call last):
>> >> >> >>       File "<string>", line 14, in <module>
>> >> >> >>       File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/build/scipy/setup.py", line 181, in <module>
>> >> >> >>         setup_package()
>> >> >> >>       File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/build/scipy/setup.py", line 131, in setup_package
>> >> >> >>         from numpy.distutils.core import setup
>> >> >> >>     ImportError: No module named numpy.distutils.core
>> >> >> >>     Complete output from command python setup.py egg_info:
>> >> >> >>     Traceback (most recent call last):
>> >> >> >>
>> >> >> >>   File "<string>", line 14, in <module>
>> >> >> >>
>> >> >> >>   File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/build/scipy/setup.py", line 181, in <module>
>> >> >> >>
>> >> >> >>     setup_package()
>> >> >> >>
>> >> >> >>   File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py25/build/scipy/setup.py", line 131, in setup_package
>> >> >> >>
>> >> >> >>     from numpy.distutils.core import setup
>> >> >> >>
>> >> >> >> ImportError: No module named numpy.distutils.core
>> >> >> >>
>> >> >> >> ----------------------------------------
>> >> >> >> Command python setup.py egg_info failed with error code 1
>> >> >> >> Storing complete log in /Users/aaronmeurer/.pip/pip.log
>> >> >> >>
>> >> >> >> [TOX] ERROR: could not install deps [numpy, scipy]
>> >> >> >> ___________________________________________________________________________________________ [tox testenv:py27] ___________________________________________________________________________________________
>> >> >> >> [TOX] ***recreating virtualenv py27 (configchange/incomplete install detected)
>> >> >> >> [TOX] emptying /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27
>> >> >> >> [TOX] /Users/aaronmeurer/Documents/python/sympy/sympy/.tox$ virtualenv --distribute --no-site-packages -p /sw/bin/python2.7 py27 >py27/log/0.log
>> >> >> >> [TOX] ***installing dependencies: numpy, scipy
>> >> >> >> [TOX] /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/log$ ../bin/pip install --download-cache=/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download numpy scipy >1.log
>> >> >> >> [TOX] ERROR: invocation failed, logfile: /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/log/1.log
>> >> >> >> [TOX] ERROR: /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/log$ ../bin/pip install --download-cache=/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download numpy scipy >1.log
>> >> >> >> Downloading/unpacking scipy
>> >> >> >>   Using download cache from /Users/aaronmeurer/Documents/python/sympy/sympy/.tox/_download/http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fs%2Fscipy%2Fscipy-0.9.0.zip
>> >> >> >>   Running setup.py egg_info for package scipy
>> >> >> >>     Traceback (most recent call last):
>> >> >> >>       File "<string>", line 14, in <module>
>> >> >> >>       File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/build/scipy/setup.py", line 181, in <module>
>> >> >> >>         setup_package()
>> >> >> >>       File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/build/scipy/setup.py", line 131, in setup_package
>> >> >> >>         from numpy.distutils.core import setup
>> >> >> >>     ImportError: No module named numpy.distutils.core
>> >> >> >>     Complete output from command python setup.py egg_info:
>> >> >> >>     Traceback (most recent call last):
>> >> >> >>
>> >> >> >>   File "<string>", line 14, in <module>
>> >> >> >>
>> >> >> >>   File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/build/scipy/setup.py", line 181, in <module>
>> >> >> >>
>> >> >> >>     setup_package()
>> >> >> >>
>> >> >> >>   File "/Users/aaronmeurer/Documents/python/sympy/sympy/.tox/py27/build/scipy/setup.py", line 131, in setup_package
>> >> >> >>
>> >> >> >>     from numpy.distutils.core import setup
>> >> >> >>
>> >> >> >> ImportError: No module named numpy.distutils.core
>> >> >> >>
>> >> >> >> ----------------------------------------
>> >> >> >> Command python setup.py egg_info failed with error code 1
>> >> >> >> Storing complete log in /Users/aaronmeurer/.pip/pip.log
>> >> >> >>
>> >> >> >> [TOX] ERROR: could not install deps [numpy, scipy]
>> >> >> >> _____________________________________________________________________________________________ [tox summary] ______________________________________________________________________________________________
>> >> >> >> [TOX] ERROR: py25: could not install deps [numpy, scipy]
>> >> >> >> [TOX] ERROR: py27: could not install deps [numpy, scipy]
>> >> >> >
>> >> >> >
>> >> >> >> _______________________________________________
>> >> >> >> testing-in-python mailing list
>> >> >> >> testing-in-python at lists.idyll.org
>> >> >> >> http://lists.idyll.org/listinfo/testing-in-python
>> >> >> >
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>



More information about the testing-in-python mailing list