[TIP] Single source bilingual namespace packages with dependencies

Barry Warsaw barry at python.org
Thu Feb 9 09:03:22 PST 2017


I'm posting this here since I discovered the problem while running tox, but it
may just be a more general packaging issue.  I submitted an issue to tox:

https://github.com/tox-dev/tox/issues/453

Let's say I have a package which lives in a namespace, supports both Python 2
and 3 from a single source, and is dependent on another package in the same
namespace.  In my case, I was porting lazr.config to Python 3.6 and it's
dependent on lazr.delegates.

Under Python 2, we need a lazr/__init__.py file to make 'lazr' a package, but
in a post-PEP 420 world, we explicitly don't want that file.  In the
lazr.config source repo, that namespace __init__.py uses the cargo-culted
mojo:

try:
    import pkg_resources
    pkg_resources.declare_namespace(__name__)
except ImportError:
    import pkgutil
    __path__ = pkgutil.extend_path(__path__, __name__)

so if you `python setup.py install` the package in a Python 2 or 3 venv,
everything works.

However, if you're in the lazr.config source repo and run `tox` (it has a very
simple tox.ini), the Python 2.7 testenv succeeds, while the Python 3 testenvs
fail because they cannot import lazr.delegates.  This confused me (and I still
don't have a complete picture, but I think I'm close ;).

In the lazr.config source repo, the `lazr` directory is at the top level, so
if you just ran `python3 -c "import lazr.config"` you'd get the one in the
source tree thanks to Python's sys.path[0] == ''.  Let's say for the moment
that in this case, you have lazr.delegates installed from your OS.  Everything
seems to work fine.

But now run `tox -e py35`.  At first glance everything looks fine.  Poke
around in the .tox/py35 venv, and you can see that lazr is a proper PEP 420
namespace package (there is no lazr/__init__.py) and both lazr.config and
lazr.delegates are installed in the venv's site-packages, with dist-info/ and
*-nspkg.pth files in site-packages, and lazr/config and lazr/delegates subdirs
there too.  Indeed, activating the tox venv and doing the imports succeeds as
you'd expect.

But `tox -e py35` fails!  It fails with an ImportError because lazr.delegates
(the dependency) cannot be imported.  I'll note that the tox.ini does not
change usedevelop from its default value, in case that matters.  Here's the
tox.ini for reference:

[tox]
envlist = py27,py34,py35,py36
skip_missing_interpreters = True


[testenv]
commands = python -s -m nose -P lazr
deps =
     nose
     coverage

What happens is that tox is importing lazr.config from the source repo
(i.e. via sys.path[0] == '') instead of from the venv's site-packages, and
since in the source repo lazr is *not* a PEP 420 namespace package (because
lazr/__init__.py exists), it cannot find lazr.delegates even though that
portion should be findable on the venv's lazr package path.

I tried several different ways to fix the problem, including adding the -s to
disable the user site directory, and -P tell nose not to modify sys.path, but
neither really solved the problem completely (although they do help).  What I
had to do was move the lazr/ directory in the source repo into a non-Python
package src/ subdirectory at the top level, with the associated changes to
setup.py and others.

Now of course, when sitting at the repo top-level, you can't just "import
lazr.config" because src/ isn't on sys.path.  Obviously easily worked around
with $PYTHONPATH, but the added benefit is that tox itself cannot bogusly
import lazr.config from the source repo, and thus via later entries on
sys.path, finds the PEP 420 namespace package.  Now that lazr/ is a namespace
package, it can find both distributions; lazr.delegates is importable and
everything's fine.

So I think this is a weird corner case because the stars have to align to
provoke it: you need a namespace package with a Python 2 workaround
__init__.py file, *and* you need a dependency in the same namespace.

What I'm not sure of is whether this is a tox bug or something more general.
I don't necessarily like having to move the source tree into a src/ subdir,
but it seems like the least intrusive solution.  In the tox issue referenced
above, I original talked about setting changedir but while that works in this
specific case, it's a suboptimal general solution as pointed out by other
comments on the issue.

Has anyone else encountered this problem?

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.idyll.org/pipermail/testing-in-python/attachments/20170209/e66f2e1a/attachment.pgp>


More information about the testing-in-python mailing list