[TIP] Single source bilingual namespace packages with dependencies

[Robert Collins]

This is the problem we touched on in distutils-sig a year or so back,
when openstack was having a headache with namespace packages - and why
we walked away from them :).

tl;dr: the two sorts of namespace packages doesn't coexist. At all.

-Rob

On 10 February 2017 at 06:03, Barry Warsaw <barry at python.org> wrote:
> I'm posting this here since I discovered the problem while running tox, but it
> may just be a more general packaging issue.  I submitted an issue to tox:
>
> https://github.com/tox-dev/tox/issues/453
>
> Let's say I have a package which lives in a namespace, supports both Python 2
> and 3 from a single source, and is dependent on another package in the same
> namespace.  In my case, I was porting lazr.config to Python 3.6 and it's
> dependent on lazr.delegates.
>
> Under Python 2, we need a lazr/__init__.py file to make 'lazr' a package, but
> in a post-PEP 420 world, we explicitly don't want that file.  In the
> lazr.config source repo, that namespace __init__.py uses the cargo-culted
> mojo:
>
> try:
>     import pkg_resources
>     pkg_resources.declare_namespace(__name__)
> except ImportError:
>     import pkgutil
>     __path__ = pkgutil.extend_path(__path__, __name__)
>
> so if you `python setup.py install` the package in a Python 2 or 3 venv,
> everything works.
>
> However, if you're in the lazr.config source repo and run `tox` (it has a very
> simple tox.ini), the Python 2.7 testenv succeeds, while the Python 3 testenvs
> fail because they cannot import lazr.delegates.  This confused me (and I still
> don't have a complete picture, but I think I'm close ;).
>
> In the lazr.config source repo, the `lazr` directory is at the top level, so
> if you just ran `python3 -c "import lazr.config"` you'd get the one in the
> source tree thanks to Python's sys.path[0] == ''.  Let's say for the moment
> that in this case, you have lazr.delegates installed from your OS.  Everything
> seems to work fine.
>
> But now run `tox -e py35`.  At first glance everything looks fine.  Poke
> around in the .tox/py35 venv, and you can see that lazr is a proper PEP 420
> namespace package (there is no lazr/__init__.py) and both lazr.config and
> lazr.delegates are installed in the venv's site-packages, with dist-info/ and
> *-nspkg.pth files in site-packages, and lazr/config and lazr/delegates subdirs
> there too.  Indeed, activating the tox venv and doing the imports succeeds as
> you'd expect.
>
> But `tox -e py35` fails!  It fails with an ImportError because lazr.delegates
> (the dependency) cannot be imported.  I'll note that the tox.ini does not
> change usedevelop from its default value, in case that matters.  Here's the
> tox.ini for reference:
>
> [tox]
> envlist = py27,py34,py35,py36
> skip_missing_interpreters = True
>
>
> [testenv]
> commands = python -s -m nose -P lazr
> deps =
>      nose
>      coverage
>
> What happens is that tox is importing lazr.config from the source repo
> (i.e. via sys.path[0] == '') instead of from the venv's site-packages, and
> since in the source repo lazr is *not* a PEP 420 namespace package (because
> lazr/__init__.py exists), it cannot find lazr.delegates even though that
> portion should be findable on the venv's lazr package path.
>
> I tried several different ways to fix the problem, including adding the -s to
> disable the user site directory, and -P tell nose not to modify sys.path, but
> neither really solved the problem completely (although they do help).  What I
> had to do was move the lazr/ directory in the source repo into a non-Python
> package src/ subdirectory at the top level, with the associated changes to
> setup.py and others.
>
> Now of course, when sitting at the repo top-level, you can't just "import
> lazr.config" because src/ isn't on sys.path.  Obviously easily worked around
> with $PYTHONPATH, but the added benefit is that tox itself cannot bogusly
> import lazr.config from the source repo, and thus via later entries on
> sys.path, finds the PEP 420 namespace package.  Now that lazr/ is a namespace
> package, it can find both distributions; lazr.delegates is importable and
> everything's fine.
>
> So I think this is a weird corner case because the stars have to align to
> provoke it: you need a namespace package with a Python 2 workaround
> __init__.py file, *and* you need a dependency in the same namespace.
>
> What I'm not sure of is whether this is a tox bug or something more general.
> I don't necessarily like having to move the source tree into a src/ subdir,
> but it seems like the least intrusive solution.  In the tox issue referenced
> above, I original talked about setting changedir but while that works in this
> specific case, it's a suboptimal general solution as pointed out by other
> comments on the issue.
>
> Has anyone else encountered this problem?
>
> Cheers,
> -Barry
>
> _______________________________________________
> testing-in-python mailing list
> testing-in-python at lists.idyll.org
> http://lists.idyll.org/listinfo/testing-in-python
>