https://pantsbuild.org/ logo
Join Slack
Powered by
Do you see the namespace packaging incantations yo...
# general
h
Do you see the namespace packaging incantations you’d expect in 
__init__.py
?
k
Library/Caches/pex/installed_wheels/04478aac471824d84810668118ad087271f7acc28f03850793592f9511290f0d/azure_monitor_opentelemetry-1.6.9-py3-none-any.whl/azure/monitor/opentelemetry
is the only nested folder in that and I don't know what I should be looking for in 
__init__.py
this is those contents
Copy code
cat __init__.py
# -------------------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License. See License in the project root for
# license information.
# -------------------------------------------------------------------------

from azure.monitor.opentelemetry._configure import configure_azure_monitor

from ._version import VERSION

__all__ = [
    "configure_azure_monitor",
]
__version__ = VERSION
h
Hmm, if there are no other 
azure
packages then that is presumably not the issue
OK, so next step is to run Pants with 
--keep-sandboxes=always
and look at the pex building command in the relevant sandbox’s 
__run.sh
And see if you can reproduce this entirely outside of Pants
OK, so, 
azure.core.tracing.ext.opentelemetry_span
is indeed not present?
Maybe 
azure.monitor.opentelemetry
incorrectly fails to declare a dep on 
azure.core.tracing
or whatever?
k
I kind of went down that route, but why does it work when I run it outside of the pex, shouldn't it just not work in both scenarios?
h
azure.core.tracing
should either be in this wheel (and it isn’t) or more likely in another azure-related wheel (and you’re saying there are none)
It should, yes
next step is to try and reproduce outside of Pants, as I mentioned above
k
reproduce the dependency issue, or reproduce it working outside of pants?
h
Reproduce the issue using just pex
that sandbox will contain the underlying pex command
which hopefully reproduces the error without all the pantsisms around it
BTW which platform is this on?
and which python?
k
Mac M1 ARM 
interpreter_constraints = ["CPython==3.12.*"]
sifting through this __run.sh not really sure what i am looking for
h
You should see a very long pex invocation
there’s a bunch of environment var stuff you can ignore
FWIW, I notice that plain 
pex azure-monitor-opentelemetry
fails where 
pip
succeeds:
Copy code
$ python3.11 -m pex azure-monitor-opentelemetry
Failed to resolve compatible distributions for 1 target:
1: cp311-cp311-macosx_14_0_arm64 interpreter at /Users/benjyw/.pyenv/versions/3.11.9/bin/python3.11 is not compatible with:
    opentelemetry-sdk 1.31.1 requires opentelemetry-api==1.31.1 but 1 incompatible dist was resolved:
        opentelemetry_api-1.33.1-py3-none-any.whl
    opentelemetry-semantic-conventions 0.52b1 requires opentelemetry-api==1.31.1 but 1 incompatible dist was resolved:
        opentelemetry_api-1.33.1-py3-none-any.whl
This may or may not be related
k
I also see references to my pex python being 
python3.9
which is different than the called out pants python. as i look though this
h
since your pex build does succeed, but then fails at runtime
Which pants version is this?
k
pants_version = "2.23.1"
h
OK, so IIRC that is running on an embedded python3.9
so that at least makes sense
k
Copy code
./pex --tmpdir .tmp --no-emit-warnings --pip-version 24.0 --python-path "<LONG PATH> --interpreter-constraint $'CPython==3.12.*' -- -c $'import hashlib, os, sys

python = os.path.realpath(sys.executable)
print(python)

hasher = hashlib.sha256()
with open(python, "rb") as fp:
  for chunk in iter(lambda: fp.read(8192), b""):
      hasher.update(chunk)
print(hasher.hexdigest())
'
im trying to find the pex command in that shell, am I on the right track here?
h
That is python code, not what I’m after
But I think you might need to update your 
pip
version, to handle that package’s deps correctly
The underlying pip version used by pex that is
not “your” pip version
Try setting
Copy code
[python]
pip_version = "latest"
👀 1
k
Okay I did that, I got excited because generate-lockfiles gave me: 
azure-monitor-opentelemetry-exporter 1.0.0b36     -->   1.0.0b37
but then same error.
they just happened to release that 12 minues ago haha
okay I ran the full pex command without pants
and I got the same error
when i then try to execute the pex
h
What is the pex command?
(redact what you must)
k
Copy code
./pex --tmpdir .tmp --jobs 10 --no-emit-warnings --pip-version latest --python-path $'<?MY_PATH?>' --output-file custom.pex --emit-warnings $'--check=warn' --requirements-pex local_dists.pex --interpreter-constraint $'CPython==3.12.*' --entry-point <SNIP>.manage $'--sources-directory=source_files' $'Flask==3.1.*' $'JPype1[numpy]==1.5.0' $'azure-monitor-opentelemetry==1.6.*' $'celery[redis]==5.4.*' $'flask-cors==5.0.*' $'fpdf2==2.8.*' $'pusher==3.*' $'pydantic==2.10.*' $'requests==2.32.*' $'types-requests==2.32.*' --lock 3rdparty/python/lockfiles/default.txt --no-pypi $'--index=<https://pypi.org/simple/>' --manylinux manylinux2014 --layout zipapp
and because I only said "pants run" worked, and I need to do a sanity check again to see if I do still get a different result and I do, here is that full command 
pants run src/python/<SNIP>/manage.py --
and the pex target is:
Copy code
pex_binary(
    name="<SNIP>-pex",
    entry_point="manage.py",
    dependencies=[
        ":manage",
    ],
)
I need to step away from a bit, but for sanity I am going to try make a repo that trys to reproduces this later tonight.
Okay I can reproduce with a clean repo: pants.toml:
Copy code
[GLOBAL]
pants_version = "2.23.1"
backend_packages = [
    "pants.backend.python",
]

[source]
root_patterns = ["src/python"]

[python]
interpreter_constraints = ["CPython==3.12.*"]
enable_resolves = true
run_against_entire_lockfile = true
default_resolve = "python-default"
pip_version = "latest"

[python.resolves]
python-default = "3rdparty/python/lockfiles/default.txt"

[python-bootstrap]
search_path = ["<PYENV>", "<PATH>", "/usr/bin/python", "/usr/bin/python3"]
src/python/manage.py
Copy code
from azure.monitor.opentelemetry import configure_azure_monitor

def main():
    configure_azure_monitor(
        logger_name="app",
    )

if __name__ == "__main__":
    main()
src/python/BUILD
Copy code
python_sources(
    name="manage",
    sources=[
        "manage.py",
    ],
)

pex_binary(
    name="the-pex",
    entry_point="manage.py",
    dependencies=[
        ":manage",
    ],
)
3rdparty/python/requirements.txt
Copy code
azure-monitor-opentelemetry==1.6.*
3rdparty/python/BUILD
Copy code
python_requirements(
    name="python-default",
    source="requirements.txt",
)
pants generate-lockfiles
Copy code
Lockfile diff: 3rdparty/python/lockfiles/default.txt [python-default]

==                      Added dependencies                      ==

  asgiref                        3.8.1
  azure-core                     1.34.0
  azure-core-tracing-opentelemetry 1.0.0b12
  azure-identity                 1.23.0
  azure-monitor-opentelemetry    1.6.9
  azure-monitor-opentelemetry-exporter 1.0.0b37
  certifi                        2025.4.26
  cffi                           1.17.1
  charset-normalizer             3.4.2
  cryptography                   45.0.3
  deprecated                     1.2.18
  fixedint                       0.1.6
  idna                           3.10
  importlib-metadata             8.6.1
  isodate                        0.7.2
  msal                           1.32.3
  msal-extensions                1.3.1
  msrest                         0.7.1
  oauthlib                       3.2.2
  opentelemetry-api              1.31.1
  opentelemetry-instrumentation  0.52b1
  opentelemetry-instrumentation-asgi 0.52b1
  opentelemetry-instrumentation-dbapi 0.52b1
  opentelemetry-instrumentation-django 0.52b1
  opentelemetry-instrumentation-fastapi 0.52b1
  opentelemetry-instrumentation-flask 0.52b1
  opentelemetry-instrumentation-psycopg2 0.52b1
  opentelemetry-instrumentation-requests 0.52b1
  opentelemetry-instrumentation-urllib 0.52b1
  opentelemetry-instrumentation-urllib3 0.52b1
  opentelemetry-instrumentation-wsgi 0.52b1
  opentelemetry-resource-detector-azure 0.1.5
  opentelemetry-sdk              1.31.1
  opentelemetry-semantic-conventions 0.52b1
  opentelemetry-util-http        0.52b1
  packaging                      25.0
  psutil                         7.0.0
  pycparser                      2.22
  pyjwt                          2.10.1
  requests                       2.32.3
  requests-oauthlib              2.0.0
  six                            1.17.0
  typing-extensions              4.13.2
  urllib3                        2.4.0
  wrapt                          1.17.2
  zipp                           3.22.0
pants package src/python:the-pex
- has the deps problem
pants run src/python/manage.py --
- has a diff problem but it can do the deps correctly
h
The 
pex
cmd line that you sent was the key
I can repro this with just
Copy code
python -m pex  --pip-version latest  --output-file custom.pex  --interpreter-constraint $'CPython==3.12.*' azure-monitor-opentelemetry==1.6.*
And then
Copy code
$ ./custom.pex 
Pex 2.40.1 hermetic environment with 1 requirement and 46 activated distributions.
Python 3.12.2 (main, Aug 30 2024, 14:24:04) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "pex", "copyright", "credits" or "license" for more information.
>>> from azure.monitor.opentelemetry._configure import configure_azure_monitor
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/benjyw/Library/Caches/pex/installed_wheels/0/3e3ae4b4f8f048df7b7056510f75dbcf41055cdd433cc9425a5b7f7d1a0163c3/azure_monitor_opentelemetry-1.6.9-py3-none-any.whl/azure/monitor/opentelemetry/__init__.py", line 7, in <module>
    from azure.monitor.opentelemetry._configure import configure_azure_monitor
  File "/Users/benjyw/Library/Caches/pex/installed_wheels/0/3e3ae4b4f8f048df7b7056510f75dbcf41055cdd433cc9425a5b7f7d1a0163c3/azure_monitor_opentelemetry-1.6.9-py3-none-any.whl/azure/monitor/opentelemetry/_configure.py", line 36, in <module>
    from azure.core.tracing.ext.opentelemetry_span import OpenTelemetrySpan
ModuleNotFoundError: No module named 'azure.core.tracing.ext.opentelemetry_span'
>>>
But:
Copy code
$ unzip -t custom.pex  | grep opentelemetry_span
    testing: .deps/azure_core_tracing_opentelemetry-1.0.0b12-py3-none-any.whl/azure/core/tracing/ext/opentelemetry_span/   OK
    testing: .deps/azure_core_tracing_opentelemetry-1.0.0b12-py3-none-any.whl/azure/core/tracing/ext/opentelemetry_span/__init__.py   OK
    testing: .deps/azure_core_tracing_opentelemetry-1.0.0b12-py3-none-any.whl/azure/core/tracing/ext/opentelemetry_span/_schema.py   OK
    testing: .deps/azure_core_tracing_opentelemetry-1.0.0b12-py3-none-any.whl/azure/core/tracing/ext/opentelemetry_span/_version.py   OK
    testing: .deps/azure_core_tracing_opentelemetry-1.0.0b12-py3-none-any.whl/azure/core/tracing/ext/opentelemetry_span/py.typed   OK
So that package is in the pex, it’s just not being loaded, and I assume this is because of namespace packaging shenanigans
Although I notice that this still fails with 
--layout packed
Gotta go now, will circle back
👀 1
g
I tried inspecting the 
azure
import from 
./custom.pex
. It looks like it has a reference to all the right wheels.
Copy code
>>> pprint(azure.__path__._path)
['/home/seb/.cache/pex/installed_wheels/0/04478/azure_monitor_opentelemetry-1.6.9-py3-none-any.whl/azure',
 '/home/seb/.cache/pex/installed_wheels/0/d44e4/azure_core-1.34.0-py3-none-any.whl/azure',
 '/home/seb/.cache/pex/installed_wheels/0/f0420/azure_core_tracing_opentelemetry-1.0.0b12-py3-none-any.whl/azure',
 '/home/seb/.cache/pex/installed_wheels/0/cabb3/azure_monitor_opentelemetry_exporter-1.0.0b37-py2.py3-none-any.whl/azure',
 '/home/seb/.cache/pex/installed_wheels/0/3ae0d/azure_identity-1.23.0-py3-none-any.whl/azure']
azure_core_tracing_opentelemetry
has the code we are trying to import, but I believe our import ends up in 
azure_core
, because it also contains the a`azure/core/tracing/ext` path:
Copy code
Directory /home/seb/.cache/pex/installed_wheels/0/f0420/azure_core_tracing_opentelemetry-1.0.0b12-py3-none-any.whl/azure/core/tracing/ext contains: ['opentelemetry_span']
Directory /home/seb/.cache/pex/installed_wheels/0/d44e4a/azure_core-1.34.0-py3-none-any.whl/azure/core/tracing/ext contains: ['__pycache__', '__init__.py']
It seems to me that python can't tell between these two and the order of paths matters. However when using 
pants run ...
Copy code
>>> pprint(azure.__path__._path)
['/home/seb/.cache/pants/named_caches/pex_root/venvs/1/s/d8411b35/venv/lib/python3.12/site-packages/azure']
and in that path it seems it has combined all the different libraries into a single azure directory
Copy code
book-builder$ ls -la /home/seb/.cache/pants/named_caches/pex_root/venvs/1/s/d8411b35/venv/lib/python3.12/site-packages | grep azure
drwxr-xr-x   5 seb seb    4096 May 30 12:32 azure
drwxr-xr-x   2 seb seb    4096 May 30 12:32 azure_core-1.34.0.dist-info
drwxr-xr-x   2 seb seb    4096 May 30 12:32 azure_core_tracing_opentelemetry-1.0.0b12.dist-info
drwxr-xr-x   2 seb seb    4096 May 30 12:32 azure_identity-1.23.0.dist-info
drwxr-xr-x   2 seb seb    4096 May 30 12:32 azure_monitor_opentelemetry-1.6.9.dist-info
drwxr-xr-x   2 seb seb    4096 May 30 12:32 azure_monitor_opentelemetry_exporter-1.0.0b37.dist-info
drwxr-xr-x   3 seb seb    4096 May 30 12:32 opentelemetry_resource_detector_azure-0.1.5.dist-info
so then my questions would be; can pex merge these separate wheels into one? should it? can we configure the python 
azure
path to guide it to the right wheel?
h
The pex needs to be built with 
--venv
and 
--venv-site-packages-copies
for this to work
that forces pex to splat itself out into a conventional venv, which overcomes various nasty behaviors that happen to work in venvs but aren’t strictly kosher
you can get Pants to supply these args with this target field for a single pex: https://www.pantsbuild.org/stable/reference/targets/pex_binaries#extra_build_args
👀 1
g
Amazing, that works. Thanks Benjy! Any side effects we should keep an eye out for when using these build args?
k
Benjy thanks so much for helping us troubleshoot this issue, really appreciate the time you spent!
h
You’re welcome.
Only side effect might be longer cold start time for the pex, since it has to rearrange itself into a venv (those venvs are cached though)
👍 1
You can measure that, perhaps
12 Views
Open in Slack
PreviousNext
Powered by