https://pantsbuild.org/ logo
#pex
Title
# pex
r

red-jackal-13334

04/23/2020, 10:35 AM
i even tried with venv, pip install azure-cli, pip freeze >reqs, pex setuptools -r reqs -m azure.cli -o az, still azure.common not found
h

happy-kitchen-89482

04/26/2020, 7:36 PM
I can repro this, digging deeper
This seems relevant: https://github.com/Azure/azure-sdk-for-python/issues/3950 although it's basically two people bickering unproductively about who said what, so maybe it's not helpful.
cc @enough-analyst-54434 who is probably the best person to tackle this
I notice that
pex azure-common setuptools  -o az
works (you can run
./az
and then
import azure.common
. But even
pex azure-cli azure-common setuptools -o az
) does not.
Digging deeper: The underlying cause appears to be related to be bad namespace package hygiene in the
azure
packages.
For background: A "namespace package" is a package spread across multiple distributions. In this case the package
azure
has subpackages in many distributions (
azure.common
,
azure.cli
,
azure.mgmt
and so on).
Unlike the JVM, which handles this sort of thing intrinsically, Python does not. The workaround is "namespace packages" as described here: https://packaging.python.org/guides/packaging-namespace-packages/
I've noticed that most azure dists don't have an
azure/__init__.py
file, so I assume the intention is that
azure
should be a native namespace package (as described at that link).
However a handful of dists do have an explicit
azure/__init__.py
(azure-cli-telemetry, azure-cli-core, azure-multiapi-storage, azure-cli, azure_cosmos) and unfortunately those
__init__.py
files contain pkg_resources-style namespace package invocations, which are not compatible with native namespace packages.
So imports from
azure
are fundamentally broken
Not sure how this works directly in
pip
, but not via pex. maybe some bad luck around ordering of imports during bootstrapping or something.
I think
azure
itself is broken, even if it somehow happens to work in practice in some cases.
Not really sure what to do about this TBH, @enough-analyst-54434 thoughts?
Looks like we have a related open issue: https://github.com/pantsbuild/pex/issues/331
This may be a common enough situation that we have to somehow fix it in pex
e

enough-analyst-54434

05/04/2020, 4:22 PM
It looks like you sussed the underlying issue. The relevant difference beteen pip and pex here is that the pip install installs all distributions in 1 chroot. As such the
azure/_init_.py
is layed down by one of these:
Copy code
$ zipinfo -1 az | grep /azure/__init__.py | while read path; do echo -e "> In $path:\n===\n$(unzip -qc az $path | grep -E -v "^#")\n==="; done
> In .deps/azure_cli-2.5.1-py3-none-any.whl/azure/__init__.py:
===
import pkg_resources
pkg_resources.declare_namespace(__name__)
===
> In .deps/azure_cli_core-2.5.1-py3-none-any.whl/azure/__init__.py:
===
import pkg_resources
pkg_resources.declare_namespace(__name__)
===
> In .deps/azure_cli_telemetry-1.0.4-py2.py3-none-any.whl/azure/__init__.py:
===
import pkg_resources
pkg_resources.declare_namespace(__name__)
===
> In .deps/azure_cosmos-3.1.2-py3-none-any.whl/azure/__init__.py:
===
__import__('pkg_resources').declare_namespace(__name__)
===
> In .deps/azure_multiapi_storage-0.3.1-py2.py3-none-any.whl/azure/__init__.py:
===
__import__('pkg_resources').declare_namespace(__name__)
===
In a pex, each dependency is installed in its own isolated chroot. As such azure-common gets no help from any of the distributuions above and does not have an
azure/_init_.py
file and so its
azure/common
contents cannot be found by any load sequence that imports
azure/*
via a pkg_resources namespace. So, that's the reason for the difference. Pip does mask bad namespace package hygiene as @happy-kitchen-89482 suggested. As for a fix for Pex for this sort of problem that does not break multiplatform PEXes (these are the reason for the isolated dependency chroots) ... it's not immediately clear to me what the best approach is.
I think the best way to handle this is to push on the
--unzip
option added recently and have PEXes execute as ~normal venv-style applications: https://github.com/pantsbuild/pex/issues/962