https://pantsbuild.org/ logo
Join Slack
Powered by
Are there any reasons that the pex generated durin...
# general
b
Are there any reasons that the pex generated during pants run, and the pex generated during pants package would be different? I am running into a weird issue where I can run package and run the pex directly and that works, but when I run pants run, I get a no module named ... error.
Copy code
ModuleNotFoundError: No module named 'azure.eventhub.extensions.checkpointstoreblobaio'
b
@brainy-sundown-66139 what are the 
pex_binary
target knobs you set? Also, can you provide the output of 
unzip -qc <your pants packaged PEX> PEX-INFO | python -mjson.tool
? The azure packages all share a namespace and namespace packages are surely at play here.
For example, to get this to work just using Pex, I need both 
--venv
and `--venv-site-packages-copies`:
Copy code
:; pex azure-eventhub-checkpointstoreblob -o example.pex
:; ./example.pex -c 'import azure.eventhub.extensions.checkpointstoreblob; print(azure.eventhub.extensions.checkpointstoreblob.__file__)'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
...
  File "-c <cmd>", line 1, in <module>
    # Copyright 2014 Pex project contributors.
ModuleNotFoundError: No module named 'azure.eventhub.extensions.checkpointstoreblob'


:; pex azure-eventhub-checkpointstoreblob -o example.pex --venv
:; ./example.pex -c 'import azure.eventhub.extensions.checkpointstoreblob; print(azure.eventhub.extensions.checkpointstoreblob.__file__)'
Traceback (most recent call last):
  File "/home/jsirois/.cache/pex/venvs/3/98c323f8246de4fdc3f1eb504232e9b7bc6cdf51/485b182d3b10432d6a5f1f5be4248d540afe6b7b/pex", line 376, in <module>
...
#!/home/jsirois/.cache/pex/venvs/3/s/1c4fc19e/venv/bin/python3.14 -sE
ModuleNotFoundError: No module named 'azure.eventhub.extensions.checkpointstoreblob'


:; pex azure-eventhub-checkpointstoreblob -o example.pex --venv --venv-site-packages-copies
:; ./example.pex -c 'import azure.eventhub.extensions.checkpointstoreblob; print(azure.eventhub.extensions.checkpointstoreblob.__file__)'
/home/jsirois/.cache/pex/venvs/3/bdb2584a84c84333339284c99cd4fcd2fc26ad74/485b182d3b10432d6a5f1f5be4248d540afe6b7b/lib/python3.14/site-packages/azure/eventhub/extensions/checkpointstoreblob/__init__.py
So, presumably, you set 
--venv
and 
--venv-site-packages-copies
in your 
pex_binary
target, but 
pants run
ignores some or all of those knobs. That's my guess anyhow.
Pants has historically traded correctness for speed when running PEXes unfortunately - that would seem to still be going on and your 
pex_binary
knobs are not being used faithfully during 
pants run
. Perhaps there is some further Pants option to make it actually obey your 
pex_binary
intent.
b
Ok yea I have narrowed it down to being a namespace issue problem but unsure what exactly could solve it. So here is my pex_binary that I had some success with (able to run via packaged pex but pants run fails)
Copy code
pex_binary(
    name="bin_macos",
    entry_point="<http://healthcloud_worker.app|healthcloud_worker.app>",
    dependencies=[":app_src"],
    resolve="python-default",
    interpreter_constraints=["CPython==3.11.*"],
    include_requirements=True,
    include_tools=True,
    execution_mode="zipapp",
    inherit_path="prefer",
)
But Ideally I wanted this configuration to work so I can use the same pex binary in my local testing and when I deploy to k8 but I had 0 success with it once I added the checkpoint package:
Copy code
pex_binary(
    name="bin",
    entry_point="<http://healthcloud_worker.app|healthcloud_worker.app>",
    dependencies=[":app_src", ":linux_x86_py311", ":macos_arm64_py311"],
    resolve="python-default",
    interpreter_constraints=["CPython==3.11.*"],
    execution_mode="venv",
    include_tools=True,
    env={"ENV": "dev"},
    complete_platforms=[
        ":linux_x86_py311",
        ":macos_arm64_py311",
    ],
)
Copy code
{
  "bootstrap_hash": "23f6f08a1ef609bba85fd8d67dd6af6ab14d1a98",
  "build_properties": {
    "pex_version": "2.55.2"
  },
  "code_hash": "60200e1944b81519fe7cc8b06e79cf8a04cd6aeb",
  "deps_are_wheel_files": false,
  "distributions": {
    "PyJWT-2.10.1-py3-none-any.whl": "17bcf7fd145ef9dade0c112aa7aab551f1425112c3ca92694a3ff237595ce91f",
    "aiohappyeyeballs-2.6.1-py3-none-any.whl": "3e656c25452c07bddc70e6bfa157f15a91c5ecac4e06b3928b36ea7c427a84a1",
    "aiohttp-3.13.2-cp311-cp311-macosx_11_0_arm64.whl": "107faf003cc03d0adc50359af56c7b177d30efb333d9b87414b499609cadf5c3",
    "aiosignal-1.4.0-py3-none-any.whl": "3dce929764358b48fe4ef668d0d963d356e0a5b5b4a98f72c54999d7b45733fd",
    "annotated_types-0.7.0-py3-none-any.whl": "b4b730bfe1091e078d59bd61d29343389775d1d8d53667ba07677070d39b44a2",
    "anyio-4.11.0-py3-none-any.whl": "05e102ed5a197a2dc430d22c12aaaa1b571b6c845cfdb4a4e2df3fb11933841d",
    "attrs-25.4.0-py3-none-any.whl": "d2885077d1fa5a3b44a9e508ec3a9c577f19dd14e36d5c076b83605512938326",
    "azure_common-1.1.28-py2.py3-none-any.whl": "79334faffb6c1a9f763f8b1357806bf6324b3d11e8b6b92a644abe8008e50a56",
    "azure_core-1.36.0-py3-none-any.whl": "32b1cb123c581c6d7d6a584c5e20b8b5009a95f7ce8e52dc40fbf84a086de1e7",
    "azure_eventhub-5.12.1-py3-none-any.whl": "214d6457862909a4875ea10a71a5890ddf4b2d8205178c1cec142683105ec38a",
    "azure_eventhub_checkpointstoreblob_aio-1.1.4-py3-none-any.whl": "f2d58512f0283074a47b3124d88be4bf0c6a663e3abc70bc1846ef39e9297452",
    "azure_identity-1.13.0-py3-none-any.whl": "05672e003ea5653c8d3ccc8a707c1e21d319f75f55243485e25582027fc55581",
    "azure_keyvault_secrets-4.7.0-py3-none-any.whl": "c9c247113d85929a16c21924a3a5de36e9af8c8917008aaa086154aecb7dcc5f",
    "azure_storage_blob-12.27.1-py3-none-any.whl": "72e4dc55399e1111086487896094f948a1c50796d5dba62b33197439d379424f",
    "certifi-2025.11.12-py3-none-any.whl": "8e7645bfc36c5f52acc57ff96b0deb1883b911b61b7bdcf4b2a036e8474aecf9",
    "cffi-2.0.0-cp311-cp311-macosx_11_0_arm64.whl": "1430d0f4e93a496b542be021c93b4a16892e2e0530b47b25adc9cfb9951accbd",
    "charset_normalizer-3.3.2-cp311-cp311-macosx_11_0_arm64.whl": "49767b7b8e0e430734387289efc66301846639cc8548b3b0e26c8c226a800ffb",
    "click-8.3.0-py3-none-any.whl": "8b694f031a64b96f45663f4e307f1b8dc9265267fee98fa4fc89c737ba1f887e",
    "cryptography-46.0.3-cp311-abi3-macosx_10_9_universal2.whl": "19cfe2e3f4364cab980f2d2d5b9ee251bb9d957e66174d2bd3973cd97bf0390c",
    "fastapi-0.115.0-py3-none-any.whl": "2440000a9dedd150db4734b2f9224a37b7cc822fce93ee3da539282ef4f045cc",
    "frozenlist-1.8.0-cp311-cp311-macosx_11_0_arm64.whl": "fd76913e74cb07696e191b05fc7857f8f6a3058ae111779693cab16b89d0f7c8",
    "h11-0.16.0-py3-none-any.whl": "49cace83b9a284604389b2f601c4b630f6996931de55932c0bad521b97e656f7",
    "idna-3.11-py3-none-any.whl": "f70adacaa2e7dd0c887073f4295857b4e844c32c052d85fe08df7efc21160884",
    "isodate-0.7.2-py3-none-any.whl": "a326b24d6f83d18ec111645a42eaa0041224caf3b8775be2c7bac1f0c8cbe906",
    "msal-1.34.0-py3-none-any.whl": "81d993903267c824970c291d28fc4691090bc579e41b82ef90c9e1983037feb0",
    "msal_extensions-1.3.1-py3-none-any.whl": "e498e498d8c0dfa0007f4df97ad4fa7889bd045aa0853c50ca4e3a8ca09c1f8c",
    "msrest-0.7.1-py3-none-any.whl": "a786ab9a7681c323e6a8418ccb3b5657e8b4c3c4756b39d13152923dd03eb988",
    "multidict-6.7.0-cp311-cp311-macosx_11_0_arm64.whl": "921e046e3217a791031c9baa4e9818aa1b5bcb2a5b6e6c0455ccc3a205321ba4",
    "oauthlib-3.3.1-py3-none-any.whl": "722ac638c907a0659538b2ce0be00b26e4db0426c58d66984086a7d34f52d204",
    "propcache-0.4.1-cp311-cp311-macosx_11_0_arm64.whl": "4c61c96b9b7969e4f41e244e651d3b11233e24f7af962f863824624d45e7cfd5",
    "pycparser-2.23-py3-none-any.whl": "8710ede061cc78b034f8eb3d76b6e1afbbed738ec659333e8d21edfe2cda862a",
    "pydantic-2.11.10-py3-none-any.whl": "213d47e6761245f23c5020230c445f9066e5d7c12444889bd2e81d8f47dbf73a",
    "pydantic_core-2.33.2-cp311-cp311-macosx_11_0_arm64.whl": "4ee1f6d5a8439d2b5629f51b799e14ca92cc18e208eb8346579e2b68c67a1157",
    "requests-2.31.0-py3-none-any.whl": "12f31dfc68f4158e5a7dc19ecdbb7aa3bec954b4dfa408c4bc6a3c3150a1037f",
    "requests_oauthlib-2.0.0-py2.py3-none-any.whl": "e2b253350a707144723810bf44fd9afad2965009fcaf178aa1e48823d92b0712",
    "six-1.17.0-py2.py3-none-any.whl": "a0ad9401c8a536207dfdd962d2a068f6960b22ea14524a8f75434f4d48b39c58",
    "sniffio-1.3.1-py3-none-any.whl": "94bbdc97078af67a4f2c999a4b1cc7c107de4ef8c0281d7c9de65a90205f40c6",
    "starlette-0.38.6-py3-none-any.whl": "ac8f28baf56eb001a189c590ba845dc7a7f18feecc3e02f53ef6ed9116c8dd34",
    "tenacity-8.2.3-py3-none-any.whl": "50e5149b5d0660f0e8d458a7d53c3320b0cf44d1171aa7ba8aeb0334e7b43f85",
    "typing_extensions-4.12.2-py3-none-any.whl": "ab87cc50d4264524df94030d479d1501922ac3d258a1d09cff936840137acd76",
    "typing_inspection-0.4.2-py3-none-any.whl": "5c1f5bcb031f1e3e264a04692129ed9ad1b3dbf49f780a4e381b8a063ac0596e",
    "urllib3-1.26.18-py2.py3-none-any.whl": "f0b2b048d0941174a2abe3ab7a6f2b48844192abdba3aaadbe83e78983387f5d",
    "uvicorn-0.30.0-py3-none-any.whl": "4cf313300d61b7665004235bd25b811606b7ee4a64d59b0277abacbb83576910",
    "yarl-1.22.0-cp311-cp311-macosx_11_0_arm64.whl": "981fe045cb6651df63614749c116b48970a7a8016020b83c8b8b7797a7232998"
  },
  "emit_warnings": true,
  "entry_point": "<http://healthcloud_worker.app|healthcloud_worker.app>",
  "excluded": [],
  "ignore_errors": false,
  "includes_tools": true,
  "inherit_path": "prefer",
  "inject_args": [],
  "inject_env": {},
  "inject_python_args": [],
  "interpreter_constraints": [
    "CPython==3.11.*"
  ],
  "max_install_jobs": 1,
  "overridden": [],
  "pex_hash": "96d53d05d1441deb1b991145892e136701f77a3f",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "azure-eventhub-checkpointstoreblob-aio==1.1.4",
    "azure-eventhub==5.12.1",
    "azure-identity==1.13.0",
    "azure-keyvault-secrets==4.7.0",
    "azure-storage-blob>=12.0.0",
    "charset-normalizer==3.3.2",
    "cryptography>=3.4.0",
    "fastapi==0.115.0",
    "pydantic==2.11.10",
    "requests==2.31.0",
    "tenacity==8.2.3",
    "urllib3==1.26.18",
    "uvicorn==0.30.0"
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_hermetic_scripts": true,
  "venv_site_packages_copies": false,
  "venv_system_site_packages": false
}
^That is for the pex binary "bin_macos"
b
According to my experiment above, neither of your 
pex_binary
targets should work. You need both 
execution_mode="venv"
and 
venv_site_packages_copies=True
.
Basically, namespace packages are luck of the draw when using zipapp execution mode.
If you inspect the 
sys.path
in such a PEX, you'll find Pex stitches together 1 
sys.path
entry per distribution instead if 1 site-packages 
sys.path
entry. Many distributions only work when installed together on 1 
sys.path
entry.
b
OK yea it seems the original one works when I add the venv_site_packages_copies=True. Thank you for the help. Yea zipapp is what gave me some success so I thought I was going in the right direction but I was probably looking in the wrong place on the docks to find the site packages copies.
b
Yeah, zipapp should have failed. You were getting lucky with sys.path entry ordering maybe? I don't buy that though. Really should have never worked.
🤷‍♂️ 1
Well, this is a highly suspect setting actually: 
inherit_path="prefer",
Why were you using that in the zipapp pex_binary?
That makes the PEX non-hermetic and it can pick up distributions from the sys.path of the Python running the PEX.
That could have allowed things to appear to work.
@brainy-sundown-66139 understanding the 
inherit_path
bit is the key here. If you did that with well-reasoned intent, that's one thing. If you were just fiddling to get things working, then that would explain things. It got things working most likely by using dists outside the PEX. That would only work "on your machine".
b
I added 
inherit_path="prefer"
because the 
pants export
venv worked in my IDE, but packaging it as a PEX kept failing with missing Azure module. That explains why it worked on my machine but failed in the Pants sandbox. I didnt have it in my initial pex binary (the one with both linux and Mac) and the other pex binary was my copy to tinker with settings while only working with local (no linux platform) but this also means that pex would never have been able to run in docker anyways and would be local only idk
b
Ok, another shrugging man defeated - excellent.
No shrugging!
👍 1
🤣 1
So, with both 
execution_mode="venv"
and 
venv_site_packages_copies=True
and appropriate platforms or complete_platforms, you should be able to have 1 
pex_binary
that works on Mac and in containers and in prod, etc. If that's what you want.
Can make for a pretty big PEX though, so you may want separate targets.
b
Yea this integration was literally just to get it done because we need data flowing, a more permanent integration is going to be built later with CI/CD pipelines and that will set up seperate pex_binaries for local dev vs container to keep things slim, but also this pex is only 26 MB. I am still learning pants though, I primarily develop in node/ typescript, but the sdks for basically all the systems we are integrating for this project are python native. Found pants looking for something that can get similar ish feel to a monorepo like pnpm.
b
Gotcha. I don't really know Pants, I'm the Pex maintainer. But hopefully your setup is stable and understood now.
👍 1
6 Views
Open in Slack
PreviousNext
Powered by