lightkube fails to import lightkube.models when wheels are on separate sys.path entries (e.g., Bazel rules_python)

[ashtuchkin]

Summary

When lightkube and lightkube-models are installed into different directories on sys.path (common with Bazel rules_python, PEX/zipapp, layered containers, or custom PYTHONPATH setups), imports inside lightkube that rely on lightkube.models fail with: ModuleNotFoundError: No module named 'lightkube.models'

How to reproduce

# Create fresh venv
python3 -m venv lk-venv
. lk-venv/bin/activate

# Install the two packages into SEPARATE directories (simulates Bazel/rules_python layout)
mkdir -p lk-repro/a lk-repro/b
pip install --target lk-repro/a 'lightkube==0.17.2'
rm -rf lk-repro/a/lightkube/{models,resources}  # Remove lightkube-models from the first directory to simulate split-layout

pip install --target lk-repro/b 'lightkube-models==1.31.1.8'

# Reproduce the failure: lightkube can't see models from the other root
PYTHONPATH=lk-repro/a:lk-repro/b python -c "import lightkube"
# Observe ModuleNotFoundError: No module named 'lightkube.models'

Root cause

• lightkube and lightkube-models both provide modules under the same top-level package name lightkube/.
• The lightkube/__init__.py in the lightkube wheel prevents implicit namespace package behavior, so Python does not merge the lightkube package tree across multiple directories.
• Thus, lightkube.__path__ points only to the first directory containing lightkube, and subpackages from the other wheel (e.g., lightkube.models, lightkube.resources) are invisible.

Proposed fix (backward-compatible, minimal)

Make lightkube a pkgutil-style namespace package by extending its package path in lightkube/__init__.py. This preserves the existing public API re-exports and works even when wheels are installed to different directories.

Add these two lines at the very top of lightkube/__init__.py:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

Why this works:

• pkgutil.extend_path tells Python to treat lightkube as a namespace spanning all lightkube/ directories on sys.path.
• It’s compatible with current packaging and does not require removing __init__.py or altering the public API.

Alternatives

• PEP 420 implicit namespace package: remove lightkube/__init__.py. Not great here because lightkube/__init__.py re-exports the public API; removing it would be a breaking change or require moving those symbols.
• Runtime workaround in user code (not ideal): mutate lightkube.__path__ or add a custom MetaPathFinder before importing lightkube to merge paths. This is what some users do under Bazel, but it’s brittle and belongs in the package.

Affected versions

• lightkube: 0.17.2 (and likely earlier)
• lightkube-models: 1.31.x
• Python: 3.12 (also reproducible on 3.10–3.11)

[ashtuchkin]

Let me know if you'd like a PR to fix this!

[gtsystem]

Hi, sure feel free to create a PR.

Do we need to do the same for the lightkube-models package?

[jonasdedden]

This idea corresponds to this solution (https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages), and would require a change in both lightkube and lightkube-models, as for this to work correctly:

Every distribution that uses the namespace package must include such an __init__.py. If any distribution does not, it will cause the namespace logic to fail and the other sub-packages will not be importable. Any additional code in init.py will be inaccessible.

(As a sanity check I implemented the change locally and it didn't work work if you don't set up the path patch in lightkube-models too)

Also note that this change is considered obsolete and deprecated:

These two methods, that were used to create namespace packages prior to PEP 420, are now considered to be obsolete and should not be used unless you need compatibility with packages already using this method. Also, pkg_resources has been deprecated.

I'd recommend leaving out the injection of models and resources entirely and move that into a separate lightkube_models package instead with its own module tree.

[jonasdedden]

Actually, the solution pkgutil-style wouldn't even work in the case of lightkube:

I played around with this example repository that's linked in the corresponding documentation from Python packaging itself.

The solution seems simple, just add the __path__ = __import__('pkgutil').extend_path(__path__, __name__) statement to __init__.py. Every package has to have this, otherwise this doesn't work. In the example, both pkg_a and pkg_b have a __init__.py with this content.

I was interested how site-packages/ would look like with both packages installed, since both pkg_a and pkg_b want to add the example_pkg/__init__.py:

$ pwd 
sample-namespace-packages/pkgutil
$ uv venv
$ uv pip install pkg_a/
$ uv pip install pkg_b/
$ ls -l .venv/lib/python3.*/site-packages/example_pkg
drwxr-xr-x a
drwxr-xr-x b
-rw-r--r-- __init__.py

This only works by non-"deterministic" overriding. The last package installed will determine the content of __init__.py:

$ echo "A = 1" >> `pkg_a/example_pkg/__init__.py`
$ uv pip install pkg_a/
$ cat .venv/lib/python3.*/site-packages/example_pkg/__init__.py
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
A = 1
$ uv pip install pkg_b/
$ cat .venv/lib/python3.*/site-packages/example_pkg/__init__.py
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
$ uv pip install pkg_a/
$ cat .venv/lib/python3.*/site-packages/example_pkg/__init__.py
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
A = 1

So this really is feature-wise really equivalent to the modern approach of "native" namespace packages where you simply have to leave out the root-level __init__.py in every package to mark them as namespace packages => no package is allowed to put helper imports there.

Particularily in the lightkube example, this wouldn't help, since there are a lot of imports at lightkube/__init__.py, and we don't want that to be un-deterministically overriden by the almost empty __init__.py of the lightkube-models's __init__.py.

Summary & Recommendation

I don't think even the pkgutil-based legacy namespace packages workaround would work in the lightkube case. As already stated, my clear & strong recommendation here would be to split up lightkube and lightkube_models into two separate module trees.

This also clearly signals to the user that lightkube is different than the other kubernetes client libraries competitors (which is a very good thing!), namely that its models are clearly distinct from the client package and can independently switched to different versions. For example, kr8s just has all their models in a single python file somewhere in the client package tree: https://github.com/kr8s-org/kr8s/blob/main/kr8s/_objects.py. kubernetes has a submodule for all models, but also thightly coupled to the client code itself since it's in the same package: https://github.com/kubernetes-client/python/tree/master/kubernetes/client/models. By splitting imports between lightkube and lightkube_models, it's trivally made clear to the user that these are separate packages. Also, it's quite common to have "plugins" packages in Python, i.e. sidecar packages which often have the pattern [basepkg]_[pluginpkgname], instead of relying onto namespace packages and linking the [pluginpkg] somewhere in the [basepkg] module tree as [basepkg].[pluginpkg].

The primariy reason why models and resources currently live in the source tree of lightkube is to allow imports such as from lightkube.resources import Pod (#117 (comment)). The same thing in principle could also be trivially achieved with separate lightkube and lightkube_models packages, and simply have a lightkube.resources module which does a from lightkube_models.resources import *. This would not break any runtime imports compared to previous lightkube package versions, but still fix the issue of @ashtuchkin and also allow local development with normal editable installs without a sidecar bash script which links modules into the source tree.