https://pantsbuild.org/ logo
#general
Title
# general
c

careful-address-89803

10/06/2022, 11:45 PM
Hey folks, I'm probably doing amending silly, but I keep getting "no module named ..." for modules that need remapping. For example, azure-identity provides azure.identity, so I have in module_mapping
"azure-identity": ["azure.identity"]
. The dependency on that package shows up in
./pants dependencies ...
, And the package shows up in the message about which packages it's putting in the pex, and the whl for that package shows up in the pex sandbox. This doesn't seem to be a problem for tests, though, as they run just fine. I'm pretty sure it has something to do with resolves, as some of them work with
./pants run
and others don't. Short of binary-searching my dependencies, is there anything I can do to debug what's going on?
1
This is with pants 2.13.0
Hnnnng it was azure-nspkg, which is a physical package that acts as a shim to namespace packages, I guess for python2 compatibility especially. But it provides a solitary empty init file, and I guess that was confusing or upsetting to pants. Removing it doesn't seem to have broken any tests. It was included to make the ansible collection azure.azcollection work, so hopefully nothing there breaks either.
e

enough-analyst-54434

10/07/2022, 12:39 AM
@careful-address-89803 Pex has several options to control how venvs are created and I expect
azure-nspkg
is one of the rare packages that is titchy wrt to how the venvs it lives in are constructed. Do you use https://www.pantsbuild.org/docs/reference-pex#venv_use_symlinks ? FWICT Pants itself uses the symlink mode for
./pants run
with no way for you to currently override this.
c

careful-address-89803

10/09/2022, 9:35 PM
we aren't using that setting, but
./pants --pex-venv-use-symlinks test ::
causes the tests to fail in the same way.
e

enough-analyst-54434

10/09/2022, 9:38 PM
Aha, ok. Thanks for testing that. So the explanation for the issue does appear to be that this is a library / set of libraries that do not, in fact, handle site-packages directories that use symlinks. It sounds like Pants needs plumbing for this toggle in more places or in 1 place that applies to more places.
It would be nice to have no toggle at all and not use symlinks, but this was added to save space and cold-start time; i.e.: it was deemed an important optimization at some point.
c

careful-address-89803

10/09/2022, 9:48 PM
makes sense. Do you want me to open an issue to track that?
e

enough-analyst-54434

10/09/2022, 9:52 PM
That would be great. I can fill in code pointer details where we miss / hardcode this later today.
c

careful-address-89803

10/10/2022, 12:12 AM
e

enough-analyst-54434

10/10/2022, 5:35 PM
To follow up, I think this was a known issue / is fixed in 2.15.0.dev3 and forward, that is if your problem targets are
pex_binary
targets with
execution_mode="venv"
set. In the course of that investigation I found this:
Copy code
PS C:\Users\John Sirois\Downloads> zipinfo azure_nspkg-2.0.0-py2.py3-none-any.whl | grep __init__
-rw-rw-rw-  2.0 fat        3 b- defN 17-Apr-10 19:06 azure/__init__.py
PS C:\Users\John Sirois\Downloads> unzip -qc azure_nspkg-2.0.0-py2.py3-none-any.whl azure/__init__.py | hexdump -C
00000000  e2 88 a9 e2 95 97 e2 94  90 0d 0a                 |...........|
0000000b
So the
azure_nspkg
__init__.py
is not empty! I guess it works but I have no clue what those bytes are except for the ending
\r\n
Windows end of line bytes.
Yeah, that particular version - and just that version - appears borked: https://github.com/Azure/azure-sdk-for-python/pull/3412 The 2.0.0rc6 uses the `pkg_resources`ns package mechanism in
azure/__init__.py
, The 2.0.0 version uses the not quite empty strange bytes above and the 3.0.0 version I linked here switches to the `pkgutil`ns package mechanism, but makes sure to restrict that to only Python<3. You might want to get off the Azure 2.0.0 train if possible?
The 2.0.0 version also doesn't have a tag. Up to 2.0.0rc6 does and 3.0.0 does. With those strange bytes 2.0.0 almost but not quite feels like a rogue employee trojan horse attempt!
c

careful-address-89803

10/10/2022, 8:45 PM
Haha yeah, we're trying to get off of that train. The ansible collection
azuru.azcollection
is the last thing holding us up on that, they haven't really been updating it. Everything seems to work fine by removing the namespace package from my requirements.
5 Views