[TIP] tox + py.test: test installed sdist

holger krekel holger at merlinux.eu
Wed Jul 6 08:11:17 PDT 2011


Hi Stony,

On Wed, Jul 06, 2011 at 15:04 +0200, Alexander Steinert wrote:
> holger krekel, Mon 2011-07-04 22:10 CEST:
> > > there one thing in tox I don't quite understand yet. Even though tox
> > > creates the sdist and installs it into the virtual env, the
> > > installed version is not used/tested if tox is started in the
> > > project's root and the source of the module/package to be tested
> > > lives there too -- which should be quite common.
> > 
> > thanks for the good report.  In order to be able to import the test.py
> > module py.test adds the parent directory to sys.path, thus the source
> > module.py is found when it is imported.  For running py.test (or
> > nosetests which does the same) standalone this sys.path hackery works.
> > 
> > Most people have their tests and package in subdirs.
> > If you put test.py into a subdir, does it work?
> 
> No. I changed my example to be less minimal and more realistic (see
> below).The result is the same. (Abs. path to) . is first in sys.path and
> ./.tox/py26/lib/python2.6/site-packages is last.

This results from your package/tests/__init__.py file which will
lead py.test to add the parent dir of package to sys.path
before importing it.  See the "Note" in this section:

    http://doc.pytest.org/en/latest/goodpractises.html?highlight=good%20practises#choosing-a-test-layout-import-rules

If you remove package/tests/__init__.py then your test will pass because
only 'package/tests' is added to sys.path in order to import test_module.

> IMHO, tox should take care that, during tests, modules are imported from
> the installed sdist. Imagine your setup.py is flaky, say you package
> no/wrong data files. You would never see that when you run tox.

I am not sure how tox could check that modules are imported from
the installed sdist.  I agree, however, that it's too easy to
test against the inlined package instead of the installed one :/

> > FWIW I once payed with tox accepting a pattern for test files
> > and copy them to a fresh directory and then run tests there.
> > This would avoid the issue you are seeing i think.
> 
> Hmm, even if I use changedir=package to let tox call py.test there,
> toxinidir is first in path :-(

Yes, that doesn't help.  If we modified py.test to append rather
than prepend to sys.path we would run into issues as well because
"package.tests.test_module" would not be present in the installed
sdist.

best,
holger


> Regards
> Stony
> 
> 
> Excerpt from the modified example:
> 
> $ hg manifest
> .hgignore
> package/__init__.py
> package/module.py
> package/tests/__init__.py
> package/tests/test_module.py
> py-1.4.3.zip
> pytest-2.0.3.zip
> setup.py
> tox.ini
> 
> $ cat setup.py
> from distutils.core import setup
> setup(
>     name = "package",
>     version = "0.1",
>     packages = ["package"],
>     )
> 
> $ cat tox.ini
> [testenv]
> commands = py.test --verbose package/tests
> deps =
>     {toxinidir}/py-1.4.3.zip
>     {toxinidir}/pytest-2.0.3.zip
> 
> $ tox --recreate -e py26
> _________________________________ [tox sdist] __________________________________
> [TOX] ***creating sdist package
> [TOX] /home/steinale/work/toxtests$ /usr/bin/python setup.py sdist --formats=zip --dist-dir .tox/dist >.tox/log/0.log
> [TOX] ***copying new sdistfile to '/home/steinale/.tox/distshare/package-0.1.zip'
> ______________________________ [tox testenv:py26] ______________________________
> [TOX] ***recreating virtualenv py26 (configchange/incomplete install detected)
> [TOX] /home/steinale/work/toxtests/.tox$ /usr/local/bin/virtualenv --distribute --no-site-packages -p /usr/bin/python2.6 py26 >py26/log/0.log
> [TOX] ***installing dependencies: /home/steinale/work/toxtests/py-1.4.3.zip, /home/steinale/work/toxtests/pytest-2.0.3.zip
> [TOX] /home/steinale/work/toxtests/.tox/py26/log$ ../bin/pip install --download-cache=/home/steinale/work/toxtests/.tox/_download /home/steinale/work/toxtests/py-1.4.3.zip /home/steinale/work/toxtests/pytest-2.0.3.zip >1.log
> [TOX] /home/steinale/work/toxtests/.tox/py26/log$ ../bin/pip install --download-cache=/home/steinale/work/toxtests/.tox/_download ../../dist/package-0.1.zip >2.log
> [TOX] /home/steinale/work/toxtests$ .tox/py26/bin/py.test --verbose package/tests
> ============================= test session starts ==============================
> platform linux2 -- Python 2.6.5 -- pytest-2.0.3 -- /home/steinale/work/toxtests/.tox/py26/bin/python2.6
> collecting ... collected 2 items
> 
> package/tests/test_module.py:5: test_member PASSED
> package/tests/test_module.py:7: test_path FAILED
> 
> =================================== FAILURES ===================================
> __________________________________ test_path ___________________________________
> 
>     def test_path():
>         pprint(sys.path)
>         pprint(module)
>         expected =  ".tox/py26/lib/python2.6/site-packages/package/module.pyc"
>         assert path.exists(expected)
>         actual = path.relpath(module.__file__)
> >       assert actual == expected
> E       assert 'package/module.pyc' == '.tox/py26/lib/pyth...package/module.pyc'
> E         - package/module.pyc
> E         + .tox/py26/lib/python2.6/site-packages/package/module.pyc
> 
> package/tests/test_module.py:13: AssertionError
> ------------------------------- Captured stdout --------------------------------
> ['/home/steinale/work/toxtests',
>  '/home/steinale/work/toxtests/.tox/py26/bin',
>  '/home/steinale/work/toxtests/.tox/py26/lib/python2.6/site-packages/distribute-0.6.19-py2.6.egg',
>  '/home/steinale/work/toxtests/.tox/py26/lib/python2.6/site-packages/pip-1.0.1-py2.6.egg',
>  '/home/steinale/work/toxtests/.tox/py26/lib/python2.6',
>  '/home/steinale/work/toxtests/.tox/py26/lib/python2.6/plat-linux2',
>  '/home/steinale/work/toxtests/.tox/py26/lib/python2.6/lib-tk',
>  '/home/steinale/work/toxtests/.tox/py26/lib/python2.6/lib-old',
>  '/home/steinale/work/toxtests/.tox/py26/lib/python2.6/lib-dynload',
>  '/usr/lib/python2.6',
>  '/usr/lib/python2.6/plat-linux2',
>  '/usr/lib/python2.6/lib-tk',
>  '/home/steinale/work/toxtests/.tox/py26/lib/python2.6/site-packages']
> <module 'package.module' from '/home/steinale/work/toxtests/package/module.pyc'>
> ====================== 1 failed, 1 passed in 0.03 seconds ======================
> [TOX] ERROR: InvocationError: '.tox/py26/bin/py.test --verbose package/tests'
> ________________________________ [tox summary] _________________________________
> [TOX] ERROR: py26: commands failed
> 
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
> 



More information about the testing-in-python mailing list