Advanced concepts

Namespace packages

Hierarchical package structure

Traditionally, Python packages were organized into a hierarchical structure with modules and subpackages being located inside the parent package directory. When submodules are imported, they are represented as attributes on the parent module. Consider the following session:

>>> import sphinx.addnodes
>>> sphinx
<module 'sphinx' from '/usr/lib/python3.8/site-packages/sphinx/'>
>>> sphinx.addnodes
<module 'sphinx.addnodes' from '/usr/lib/python3.8/site-packages/sphinx/'>

This works fine most of the time. However, it start being problematic when multiple Gentoo packages install parts of the same top-level package. This may happen e.g. with some plugin layouts where plugins are installed inside the package. More commonly, it happens when upstream wishes all their packages to start with a common component.

This is the case with Zope framework. Different Zope packages share common zope top-level package. dev-python/zope-interface installs into zope.interface, dev-python/zope-event into zope.event. For this to work using the hierarchical layout, a common package has to install zope/, then other Zope packages have to depend on it and install sub-packages inside that directory. As far as installed packages are concerned, this is entirely doable.

The real problem happens when we wish to test a freshly built package that depends on an installed package. In that case, Python imports zope from build directory that contains only zope.interface. It will not be able to import zope.event that is installed in system package directory:

>>> import zope.interface
>>> zope
<module 'zope' from '/tmp/portage/dev-python/zope-interface-4.7.1/work/zope.interface-4.7.1-python3_8/lib/zope/'>
>>> zope.interface
<module 'zope.interface' from '/tmp/portage/dev-python/zope-interface-4.7.1/work/zope.interface-4.7.1-python3_8/lib/zope/interface/'>
>>> import zope.event
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'zope.event'

Now, this could be worked around by copying all other subpackages back to the build directory. However, there is a better solution.

Namespace package structure

Unlike traditional packages, namespace packages act as a kind of proxy. They are not strictly bound to the containing directory, and instead permit loading subpackages from all directories found in module search path. If we make zope a namespace package, we can import both the locally built zope.interface and system zope.event packages:

>>> import zope.interface
>>> import zope.event
>>> zope
<module 'zope' (namespace)>
>>> zope.interface
<module 'zope.interface' from '/tmp/portage/dev-python/zope-interface-4.7.1/work/zope.interface-4.7.1-python3_8/lib/zope/interface/'>
>>> zope.event
<module 'zope.event' from '/usr/lib/python3.8/site-packages/zope/event/'>

There are three common methods of creating namespace packages:

  1. PEP-0420 namespaces implemented in Python 3.3 and newer,

  2. Using pkgutil standard library module,

  3. Using namespace package support in setuptools (discouraged).

PEP-0420 namespaces are created implicitly when a package directory does not contain file. While earlier versions of Python (including Python 2.7) ignored such directories and did not permit importing Python modules within them, Python 3.3 imports such directories as namespace packages.

pkgutil namespaces use with the following content:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

setuptools namespace can use with the following content:


Alternatively, setuptools normally installs a .pth file that is automatically loaded by Python and implicitly injects the namespace into Python.

Both pkgutil and setuptools namespaces are portable to all versions of Python.

PEP-0420 and pkgutil namespaces are considered mutually compatible, while setuptools namespaces are considered incompatible with them. It is recommended not to mix different methods within a single namespace.

More general information on the topic can be found under packaging namespace packages in Python Packaging User Guide.

Namespace packages in Python 3

Since all supported Python versions in Gentoo support PEP-0420 namespaces, the other two methods are technically unnecessary. However, the incompatibility between pkg_resources namespaces and the other two methods makes removing them non-trivial.

If all packages within the namespace are using only pkgutil-style namespaces, you can safely remove the dependencies on the package providing the namespace and the package itself. Even partial removal should not cause any issues. However, if the package was explicitly provided upstream, note that some packages may carry an explicit dependency on it and that dependency would need to be removed or made conditional to Python < 3.3. You will also need to strip colliding files.

If setuptools-style namespace are used, the namespace packages need to remain as-is for the time being, as otherwise tests relying on the namespaced packages are going to be broken. We have not yet conceived a way forward for them.

Packaging pkgutil-style namespaces in Gentoo

Normally all packages using the same pkgutil-style namespace install its file causing package collisions. As having this file is no longer necessary for Python 3.3 and newer, the recommended solution is to strip it before installing the package. The presence of this file is harmless during build and testing.

python_install() {
    rm "${BUILD_DIR}"/lib/jaraco/ || die

Packaging setuptools-style namespaces in Gentoo

Similar approach is used for setuptools-style namespace packages. The only differences are in code and removal method.

The dev-python/namespace-<name> package for setuptools-style namespace should use the following code:

 # Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2


 PYTHON_COMPAT=( pypy3 python{2_7,3_{6,7,8}} )
 inherit python-r1

 DESCRIPTION="Namespace package declaration for zope"

 KEYWORDS="~alpha amd64 arm arm64 hppa ia64 ~m68k ~mips ppc ppc64 s390 ~sh sparc x86 ~amd64-linux ~x86-linux ~ppc-macos ~x64-macos ~x86-macos ~sparc-solaris ~sparc64-solaris ~x64-solaris ~x86-solaris"


 src_unpack() {
     mkdir -p "${S}"/zope || die
     cat > "${S}"/zope/ <<-EOF || die

 src_install() {
     python_foreach_impl python_domodule zope

Setuptools normally do not install files but *.pth files that do not collide. It is therefore easy to miss them but they can cause quite a mayhem. Therefore, remember to strip them:

python_install_all() {
    find "${D}" -name '*.pth' -delete || die